# Hill-Climbing on an Eval

> **If you arrived via `/claude-api hillclimb`:** this is the right file. Work through the steps in order - Steps 0 and 0.5 are hard prerequisites; Steps 1-2 feed the plan sign-off you need before the loop starts. Don't summarize this guide; execute it.

This guide is for iteratively improving a Claude-powered app against a fixed eval: run the eval, read the failures, change something in the codebase, run again, and repeat until the score stops moving or the budget runs out. It picks up where eval-building leaves off - the user has a way to measure; now they want to move a number: usually the quality score up, but just as often cost or latency down while quality holds.

The loop itself is simple. What makes it work or not is discipline: reading the actual transcripts rather than pattern-matching on summary stats, knowing which change produced which effect, keeping a clean record of what was tried, and not fooling yourself by tuning on the same cases you score on. **The number that matters at the end is the test-set delta versus the starting point** - improvement on the cases you read while iterating is not the result, it's the process. Those are the things this guide is opinionated about. Everything else - what to change, how many rounds to run, when to stop - is the user's call, and you should ask rather than assume.

Stay recommendation-forward - propose a concrete default with every question so the user can just say "yes" - and treat the plan sign-off at the end of Step 2 as the minimum approval you need before the loop starts. How often to check in during the loop is one of the Step 2 questions; don't ask it separately here.

> **Talking to the user.** These steps are your execution plan, not a script to narrate. Keep user-facing messages short and outcome-focused: what you ran, the score, what you'll try next, a path or link to open. Don't walk the user through which step you're on, which files you're writing, or internal bookkeeping unless they ask. One concise update per round is enough; put detail in `report.html` and `narrative.md`, not the chat. When you need a decision - what's in scope to change, which change to try next, whether to spend another round - use the `AskUserQuestion` tool rather than free-text prose: batch up to four related questions into one call, give each two to four concrete options with your recommendation listed first and labelled "(Recommended)", and don't add your own "Other" option - the tool appends a free-text one automatically. If `AskUserQuestion` isn't available (headless runs), fall back to one short question at a time.

> **Run the eval command in the background; keep the conversation free.** An eval run can take minutes to hours - don't make the user sit through it, but don't wrap it in a subagent either. Each round, do the quick parts yourself in the main session - apply the change, write `vN/change.*` - then launch the runner as **one `Bash` call with `run_in_background: true`** (the eval command itself, not an `Agent`). You'll get a completion notification when it exits; meanwhile stay available for the user's questions and, when useful, spawn the analyzer subagent (Step 4) - the only subagent this loop uses. When the runner finishes, **verify on disk before trusting the notification**: `vN/results.jsonl` should have N×R rows and `vN/summary.json` should exist; if rows are short, re-launch the same command (resume is idempotent at (case, rep), so it picks up where it stopped). Then regenerate `report.html` via the report builder (full or lite, per build-eval.md), tell the user the score and the path, and pick the next change - via `AskUserQuestion` if the Step 2 cadence has you checking in now, otherwise just state it and proceed. Before the first unattended round, show the user the exact runner command and ask them to allow it for the session, so a permission prompt can't stall a round nobody is watching. That approval covers the harness as it stands: the scaffold runner hashes itself plus `_state.json.harness_paths` and exits 2 when the hash differs from the one last recorded with `--approve-harness` - so a round that edits any harness path stops the next run until the user has seen the diff and said OK. Never pass `--approve-harness` from an unattended round; it is the user's to run. If you're running headless (`-p` / SDK) there's no conversation to keep free - run the eval in the foreground instead. When the user asks how a run is going, answer from the runner's own progress line (the scaffold prints `k/N done ... ~Ns left` every 30 s and mirrors it to `vN/progress.txt`) - one line, not a narration.

---

## Step 0: Confirm there's a runnable eval

Ask the user:

> Do you have an eval script for this flow - something I can run from the command line that exercises the app against a fixed set of prompts and prints a score?

If yes, ask for the command and where it writes its per-case results. Also ask whether the runner retries failed cases - and if so, which attempt's grade, transcript, and usage land in the results; score strict per attempt (a case that passed only on retry is a fail unless the user decides otherwise) and make sure retry attempts show up in cost accounting rather than being silently absorbed. Make sure the eval measures the outcome you're trying to improve, not just a behavior you assume correlates with it. If you're using a behavioral proxy because the real outcome is too expensive to measure every round, say so up front and run the winner on the real eval once at the end. **Each eval run must capture, per case: the full transcript, `model`, `usage` (input/output/cache token counts), and the grade dict** - not just an aggregate score. The transcript and the score for a case must come from the *same* model call; don't re-run the model separately to collect a transcript and then grade a different sample. Search the user's codebase for an existing runner or wrapper for this eval that already persists those fields before building anything new. Run it once to confirm it works and see the output shape (or work from a recent results file if one's handy). Then **read `shared/evals/eval-audit.md` and run it against the eval** - cases, runner, grader - reporting per its §6; an eval that wasn't built by `build-eval` hasn't had these checks, and Step 0.5 below is the must-pass subset, not the whole list.

If no, **stop here**. Hill-climbing without an eval is just editing and hoping. Route the user to `/claude-api build-eval` (read `shared/evals/build-eval.md` and run that flow), and come back when there's a script and a baseline number.

If the reason for hill-climbing is a model migration - the user wants to move to a newer Claude model and tune their prompts for it - read `shared/model-migration.md` alongside this guide so your proposed changes account for the new model's breaking changes and behavioral shifts.

---

## Step 0.5: Prove the eval can be climbed

A runnable eval (Step 0) is not yet a *trustworthy* one. Before you spend a round, rule out the possibility that the harness is lying to you - a hill-climb on a broken measurement is worse than none, because you'll "improve" an artifact, declare victory, and ship nothing. `eval-audit.md` (loaded in Step 0, or by `build-eval` if that's how the eval was made) is the full checklist; the checks below are the ones that must pass before round 1, each of which has silently wrecked a run:

- **Prove the eval can detect the win you're after.** From the baseline at its actual rep count, put three numbers in front of the user: the **noise floor** on the Step 1 target (paired-difference CI half-width at the current n × R - `eval-audit.md` §5 has the arithmetic), the **headroom** (ceiling minus baseline), and the **smallest improvement they'd act on**. If the noise floor is bigger than either, the loop cannot show a real in-scope win no matter how good the changes are - say so now, not after five rounds, and offer more reps, more cases, or a finer-grained metric before starting.
- **Prove the mechanism is actually wired.** Whatever the score depends on - a memory store the agent writes to, a tool it should call, a file it should read - run a one-off probe that it takes effect end-to-end before you trust any score: write a value and read it back through the same path the eval uses, or confirm the tool actually shows up in the agent's tool list. If the eval comes back as though the mechanism does nothing, check the wiring before concluding "the model can't do this."
- **Recompute the headline number from raw per-case results.** Don't trust an aggregate field in a manifest - recompute the number you'll report from the per-instance values in `results.jsonl` yourself. Mean-vs-sum and similar aggregation mixups produce spectacular phantom results that look exactly like a breakthrough until you hand-check them. A too-good-to-be-true number is a measurement bug until a manual cross-check says otherwise; make the cross-check a step, not a lucky catch.
- **Spot-check the grading on the baseline failures.** For a handful of the lowest-scoring baseline cases, read the model's actual output, the judge's reasoning, and the expected value: did the judge grade fairly, and is the ground truth correct? A wrong rubric or wrong expected value will send every round chasing a harness fix for a measurement error. If you find one, fix the rubric/GT and re-grade the baseline in place (re-run the judge on the stored transcripts - no model re-run needed) before round 1. While you're there, triage *every* zero-scoring baseline case: classify each as harness error vs. grader verdict (read the per-case error field or errors sidecar, wherever the runner records failures), and exclude the harness-error cases from the scored denominator before round 1. Spot-check the graders your gates depend on: confirm each reads the artifact the agent actually writes, and that nothing outside the fixture can flip it (host repo state, pre-seeded files, wall clock) - a grader that escapes its fixture measures the environment, not the agent.
- **Verify what served the requests and how the runner retries.** Run one smoke case and read `model` from the *response*, not your config - silent server-side substitution invalidates every comparison - and confirm request retries back off with jitter and are counted, not absorbed. `runner-scaffold.mjs` asserts both by default; a user-supplied runner needs the check (`eval-audit.md` §2-3).

- **If the artifact you score is generated from the artifact you tune, measure its build variance first.** Some flows put a stochastic generation step between the lever and the score: the prompt you're iterating on *builds* something - a memory store, a retrieval index, a synthesized corpus - and the eval then scores reads against the built thing. When that build runs once per variant and every rep reads the same build, reps and their CIs measure only the noise of scoring a fixed build; the build's own run-to-run variance is sampled once per variant, invisible to every gate, and can be the larger term. Before round 1, rebuild the baseline artifact two or three times with the prompt *unchanged* and score each build the same way: the spread across those no-change rebuilds is the floor a one-edit effect has to clear. In one climb, three builds of the same prompt spanned ~7 points against rescore noise near ±1.4 on the train mean - every edit had been compared against a single baseline build, and the loop could not tell any of them from the default. If the build spread exceeds a plausible one-edit effect, build K times per variant and compare build-pooled means, or move the lever closer to the score; adding reps over one build can't see it.

If the flow spawns subagents, also confirm the parameters you're iterating on - model, effort, prompt - actually reach every subagent and that traces capture their turns; a knob that silently doesn't propagate makes every round on it a no-op.

If you have a choice of which eval or slice to climb on, **pick the one with the most signal per token**:

- **The mechanism must actually drive the score** - disable it and re-run; if the score barely drops, the eval isn't measuring what you're tuning, and no amount of tuning will show up.
- **Low run-to-run variance at small rep counts** - if the baseline's per-rep scores swing widely, a "win" is indistinguishable from variance. Raise reps or pick a calmer slice rather than chasing noise.
- **An inspectable mechanism** - prefer an eval where you can *see why* a variant won (an artifact it wrote and reused, a tool-call trace) over a black-box delta you can't attribute and that may not generalize.

---

## Step 1: Agree on the goal, what to change, and how it's wired in

**First, the goal.** "Make the number go up" is only one of the things a hillclimb is for, and the loop behaves differently for each - so always ask before anything else, via `AskUserQuestion`, listing the metrics and perf fields the eval actually records:

> What should this hillclimb optimize?
> - **Raise `<headline metric>`** (Recommended when the eval is new and the score has obvious room)
> - **Cut cost per request** - hold `<headline metric>` within noise of baseline
> - **Cut latency** - hold `<headline metric>` within noise of baseline
> - **Move to `<other model>`** and recover `<headline metric>` on it

The answer sets three things for the rest of the loop: the **primary target** the analyzer is pointed at each round (Step 4), the metric the **stopping condition** is phrased against (Step 2), and the **guardrails** - every other recorded metric becomes a must-not-regress-outside-noise constraint rather than something to improve. If the goal is cost or latency, make sure `cost_usd` / `latency_s` is in `_state.json`'s `perf_fields` whether or not it was picked as a display column in build-eval - the goal forces the column. Record the goal in `_state.json` (e.g. `"approve_each_round": false,
  "goal": {"target": "cost_usd", "direction": "lower", "hold": ["accuracy"]}`) so a resumed session doesn't silently revert to climbing the score.

Then ask which part of the app is on the table:

> What do you want me to iterate on? For example:
> - The system prompt
> - A specific skill or instruction file the agent reads
> - Tool descriptions
> - Model choice or API parameters (`effort`, `thinking`, `max_tokens`)
> - The agent loop / harness code itself
> - All of the above - whatever moves the target
>
> And is there anything that's off-limits - parts of the prompt or code I should leave alone even if I think changing them would help?

Record the answer. This defines what you'll be editing each round; the off-limits list is a hard constraint. Also have the user name the **harness paths** - the runner script, grader, and any file the eval command executes - and record them in `_state.json.harness_paths` (repo-relative); the runner refuses to start when any of them changed since the user last ran it with `--approve-harness`, which is what keeps a harness edit from executing unreviewed in an unattended round. If the user says "whatever moves it," that's fine - but still ask about off-limits, because there's almost always something (a compliance disclaimer, a tone requirement, a tool that's contractually required). For a cost or latency goal, model choice and API parameters (`effort`, `max_tokens`, caching) are usually the biggest levers - make sure they're explicitly in or out.

If the target is a skill or instruction file, confirm **how it reaches the model during the eval**: is it appended directly into the system prompt (which isolates "is the content good?"), or loaded through the app's real skill-discovery path (which also tests "does the model find and use it?")? Both are valid and they measure different things - ask which one the user wants, and make sure the eval runner matches.

Two things you can choose freely unless the user objects: the model that *reads transcripts and proposes changes* doesn't have to be the model under test - using a stronger model for diagnosis is often worth it - and the proposed changes don't have to be prose. If a concrete helper script, a code snippet, or a worked example would guide the model better than another paragraph of instructions, write that instead.

---

## Step 2: Agree on a stopping condition (and a budget, if cost matters)

Ask via `AskUserQuestion` how many rounds to run before checking back in:

> One pass over the full set is N cases × R reps on `<model>`, roughly ~Y minutes. How many rounds before I check back in?
> - **Until plateau (Recommended):** keep going until the Step 1 target improves by less than delta for K consecutive rounds (K >= 3 - two flat rounds is too few to call a plateau), or a guardrail metric regresses outside noise, then report.
> - **One round at a time:** propose, run, report, ask again. Pick this to steer each change.
> - **N rounds:** run N, report, ask whether to continue. (User types N.)

This is both the stopping condition and the check-in cadence - the loop runs autonomously between check-ins.

Set delta above the noise floor from Step 0.5 - a stopping threshold finer than the eval can resolve never fires honestly.

**If reducing cost is itself the Step 1 goal** - not just a ceiling on this loop - read `shared/evals/cost-hillclimb.md` before planning rounds: it narrows this guide's loop to the cost objective, with a lever search order (caching health -> prompt audit -> a model × effort staircase walk -> prompt climb on the frozen model -> a down-left re-probe -> a registered joint confirm), pre-registered adoption gates, and cost-specific stopping rules. (`shared/cost-optimization.md` is the no-eval checklist for the same goal; inside this loop, follow `cost-hillclimb.md`.)

**If the user asks what the loop will cost or gives you a budget**, also present the **total loop cost** - baseline plus every planned round - and get a ceiling. Estimate it from real numbers, don't guess:

1. From a recent results file (or a small sample run if none exists), sum the `usage` fields across all cases, including judge calls if model-graded.
2. Multiply by the per-token prices for the user's provider - the Current Models table in `SKILL.md` is first-party; ask or look up if they're on Bedrock/Vertex/etc. Cached reads are ~10× cheaper than base input. This gives the cost of one pass at one rep.
3. Measure wall-clock for that pass - time a real run end-to-end and scale; don't estimate.
4. Add a rough allowance for the per-round analysis turns - typically small relative to the eval itself.

Present `(N + 1) × R × $X` and ask what total spend they're comfortable with. If they hesitate, offer the levers: fewer rounds, fewer reps, a cheaper judge model, or trim to the discriminating cases (rank by cross-rep variance from the baseline run, keep the top K, then run the full set on baseline + winner at the end to confirm). Whatever they pick, the default inside the loop is still to run the **same** set every round; never silently subset it to fit. **The N, R, and case count you present must be what you actually run** - if reps/split later push the total above the approved ceiling, mention it before proceeding.

When the goal is cost and the candidate change is prompt text, price the instruction itself first - its per-request input tokens × requests per case, against the predicted saving; some candidates disqualify on paper before you spend a round.

### Get the plan approved

Before you touch any files, confirm the plan with the user and get a clear yes. Two pieces:

- **A scope table** - two columns, "will change" and "won't touch," populated from Step 1. The user should be able to glance at it and know exactly which files and knobs are in play.
- **Who applies changes** - by default you apply each round's change and the user reviews the result; offer the alternative of **showing each round's diff for a yes/no before it runs** (recommend it when the artifact is customer-facing copy, legal/medical/regulated content, or anything the user said a human must own). Record the choice as `_state.json.approve_each_round`.
- **A short approach paragraph** - how you intend to run the loop. For example: *"Each round I'll have a fresh analyzer read the train traces and propose one change; I'll apply it, rerun the full set, and post a status table. Early rounds I'll steer toward different levers (tool descriptions, system-prompt wording, `effort`) to find where the headroom is. Running 8 rounds max."*

Don't make the user ask for this; produce it by default. It's what lets them trust the loop enough to let it run without checking in every twenty minutes.

---

## Step 3: Set up state, split the data, and take a baseline

The loop will run for multiple rounds, possibly across multiple sessions. Keep state on disk so it survives interruption and so the user can see the history. If the user already has a results directory and file layout from a prior eval, **keep theirs** - what matters is that each run produces the per-case data from Step 0 (full transcript, `model`, `usage`, grade dict, all from the same model call); the layout below is the default when starting fresh. Otherwise, create a working directory - `.claude/hillclimb/<flow-name>/` is a reasonable default, but put it wherever fits their repo - with this layout. **The report generator reads this tree directly**, so the file names and field names below are an exact contract, not a suggestion:

```
.claude/hillclimb/<flow>/
  _state.json            # loop state + report config - shape below
  metrics.md             # free-text rubric (markdown) - what each metric means; the full viewer renders it
  narrative.md           # model-authored running exec summary - rewritten after every round; final version in Step 5
  trajectory/
    scores.tsv           # derived by the report builder (full or lite): per-case mean of the primary metric, one column per round
  baseline/
    results.jsonl        # one JSON object per (case, rep) - shape below
    summary.json         # per-variant header - shape below
    traces/
      <id>_rep<k>.json   # full transcript, one file per rep (train split only)
  v1/
    change.md            # what changed this round and why (from the analyzer)
    change.patch         # the actual diff applied to the codebase
    results.jsonl  summary.json  traces/<id>_rep<k>.json
  v2/
    ...
```

**`_state.json`** - everything here is optional except the split ids; `metrics` and `perf_fields` let you override what the report infers from the data. `best.round` is 0-indexed (0 = baseline). If the eval has multiple metrics, **the first `binary`-kind metric (else `metrics[0]`) is the report's headline** - order the list accordingly. `goal` records the Step 1 answer - the metric or perf field the loop is optimizing, its direction, and the metrics it must hold - so `best` is picked against the goal, not blindly against the headline:

```json
{ "current_round": 2, "reps": 2,
  "goal": {"target": "pass", "direction": "higher", "hold": ["cost_usd"]},
  "approve_each_round": false,
  "best": {"round": 1, "test_score": 0.70},
  "train_ids": ["case_01", ...], "test_ids": [...],
  "harness_paths": ["eval/run-eval.mjs", "eval/grade.mjs"],
  "harness_sha": "...written by the runner on --approve-harness; never edit by hand...",
  "metrics": [
    {"id": "pass",      "kind": "binary", "label": "Pass"},
    {"id": "quality",   "kind": "judge"},
    {"id": "verbosity", "kind": "float",  "better": "lower"}
  ],
  "perf_fields": [
    {"id": "cost_usd",  "label": "Cost",    "unit": "$"},
    {"id": "latency_s", "label": "Latency", "unit": "s"}
  ],
  "prices": { "my-custom-model": {"in": 2.0, "out": 8.0} } }
```

**`results.jsonl`** - one line per case per rep, **appended as each case completes** so a crash doesn't lose finished work. `grade` can be a bool, a number, or a `{metric_id: number}` dict; `explanation` is an optional `{metric_id: "judge reasoning"}` sibling for judge-kind metrics. Put the full prompt text in `prompt` (the report shows it). `tags` is ordered - `tags[0]` is the primary grouping key. Record `model` from the response, not from config - `cost_usd` is derived from each row's `model` × `usage` (plus `judge_model` × `judge_usage` when present), so a model swap can't carry a stale rate and the runner doesn't compute cost at all. The full viewer does that derivation when it is on disk; otherwise compute `cost_usd` per row yourself when you report, with the recipe in build-eval.md § Before the first paid call (the "If the user asks what this will cost" bullets: Current Models prices in `SKILL.md`; cache writes at 1.25× input, cache reads at 0.1× input) - that is where the status table's `$/run` and `spend` come from. The adapter is forgiving on input: `prompt_id` may also be spelled `id` or `case_id`, and if `_state.json` omits `metrics` they're inferred from the union of grade keys. Perf keys are read by exact name:

```json
{ "prompt_id": "case_17", "rep": 0, "prompt": "...full prompt text...",
  "tags": ["topic-a", "hard"],
  "grade": {"pass": 1, "quality": 7.2, "verbosity": 3.0},
  "explanation": {"quality": "Cites two sources; balanced."},
  "model": "claude-opus-5",
  "latency_s": 12.4,
  "tool_calls": 2, "web_searches": 1,
  "usage": {"input_tokens": 1200, "output_tokens": 480} }
```

Two more optional per-row keys the report understands: `attachments` (a list of `{kind: "image"|"file"|"url", ref, alt?}` shown alongside the prompt) and `meta` (an arbitrary dict surfaced in the transcript header).

**`summary.json`** - per-variant header for the report, not aggregate stats (the report recomputes those from `results.jsonl`). All optional; `target` is one of `"system_prompt" | "skill" | "tools" | "code"`:

```json
{ "description": "Enable web_search tool",
  "target": "system_prompt",
  "suspicious": "val lift not replicated on a second seed" }
```

**`traces/<id>_rep<k>.json`** - the verbatim conversation as a JSON list of `{role, content, thinking?, name?}` turns where `role` is `system | user | assistant | tool_call | tool_result`.

The `traces/` directories are what the analyzer reads between rounds. Keeping them per-round with one file per rep means that by round 3 the analyzer can diff `baseline/traces/case_17_rep0.json` against `v2/traces/case_17_rep0.json` and see exactly what behavior changed. `trajectory/scores.tsv` is the at-a-glance cross-round view: one row per case, one column per round, cell = that case's mean primary-metric score - derived by the report builder (full or lite, identically) from the `results.jsonl` files, so the runner doesn't write it and it can't drift.

**Make artifacts visible.** If the flow consumes or produces visual or structured artifacts - input images or PDFs, computer-use screenshots, generated HTML/SVG/plots, files the model wrote - make sure a human reviewing a case can actually see them, not just a filename - rendered in the Transcripts tab with the full viewer, opened from the referenced path otherwise. The full viewer has prebuilt slots for this; the runner just fills them:

- **Input artifacts** - on the `results.jsonl` row: `"attachments": [{"kind":"pdf","ref":"baseline/inputs/case_3.pdf","alt":"source doc"}]`. Renders above the first user turn.
- **Output artifacts** - on the trace turn that produced them: `{"role":"assistant","content":"...","attachments":[{"kind":"html","ref":"v2/out/case_3.html"}]}`. Renders below that turn. Write the artifact to disk under the variant dir and `ref` it relative to the flow root.
- **Rich content in the response text** - fenced ` ```html `, ` ```svg `, ` ```json ` blocks in an assistant turn's `content` get a "> Render" toggle automatically; nothing extra to write.

The full viewer renders `image`/`svg` inline, `html` in a sandboxed scrollable iframe, `pdf` in the browser's native viewer, `json`/`text` in a `<pre>` - each with a Hide/Show toggle. Anything else (`file`: docx, pptx, ...) shows as a download chip. Paths under ~2 MB are inlined into `report.html`; larger ones stay as download links so the report doesn't balloon. **One slot per artifact**: when a turn has `attachments`, the viewer suppresses its inline-render toggle for fenced blocks in that turn's text - so put the output in `attachments` once and the response text stays plain source. The lite report renders none of this - it links each trace file, and the analyzer and the user open the `ref`'d files directly - so keep every `ref` relative to the flow root either way. If a flow needs something the prebuilt viewer doesn't cover, edit `report/frontend/atoms.jsx` (`ArtifactView`) and rebuild via `frontend/scripts/build.sh` (EAP install only - the CLI doesn't extract the frontend).

**Split the prompt set so the analyzer can't overfit to the number you report.** The analyzer reads transcripts to propose changes - it will, by design, fix the specific cases it sees. The score you report must come from cases it never read. How you carve that depends on how many cases you have; pick the lightest structure that gives a held-out number you can trust:

- **Default - train / test.** *Train* is the set whose transcripts the analyzer reads each round. *Test* is everything else: scored every round alongside train, never opened by the analyzer; its score picks the winning round and is the headline. **Draw the split at random, stratified by `tags[0]` - never by baseline score.** Train needs enough failures to show a pattern (a handful to a couple dozen), but get them by making train big enough, not by hand-picking the worst cases: a train slice selected for low scores means the analyzer only ever sees pathological cases and tunes for the tail, and those cases regress toward the mean on re-run anyway - a healthy train gain with a flat test set is the signature. After the baseline, check that train and test means agree within noise; if they don't, re-draw before round 1. The analyzer can still *focus* on the failures within train.
- **Small set, or a cross-case metric** (pairwise ordering, ranking - anything that fragments inside a slice). Don't split. Score the whole set every round, lean on **reps** to tighten the noise, and have the analyzer read a few targeted failure transcripts rather than a fixed slice. Label per-round scores in the report as **directional**: an improvement that holds across reps is real signal, but with no held-out set the headline is an iterate-on number, not a publish number.
- **Large set** (~150+ cases) where you want a final number untouched by round selection: optionally carve a third *validation* slice - scored each round to pick the winner - and hold *test* back until the end. For most evals the extra bookkeeping isn't worth it.

Be honest with the user about what their split can and can't tell them. For a binary pass-rate the 95% CI half-width is roughly `1/sqrt(n·reps)` - 25 test cases at 2 reps is about ±14 points, 50 at 2 reps about ±10 - so show that number for their set and let them choose; reps and test-size are two knobs on the same dial. Whatever they pick, record the IDs in `_state.json` exactly as they appear in `baseline/results.jsonl`'s `prompt_id` (a runner that sanitizes ids for file paths keys its rows on the sanitized form), fix the split once, and don't change it.

**Reps per prompt** is the other knob. Model outputs vary run-to-run, and repeating each prompt tightens the estimate - especially worth it on a small set, or whenever the Step 2 budget has room. Offer it and let the user pick the count; if they choose more than one, record results as "k of N reps passed" rather than a single pass/fail. Don't default this silently in either direction. Reps tighten only the noise they re-sample - a runner that reuses a once-built generated artifact across reps leaves its build variance untouched (see the Step 0.5 rebuild check).

**Isolate ground truth from the model under test.** If the eval has reference answers, rubrics, or expected outputs, make sure they are **not reachable from the model's context** - not in its system prompt, not in a tool result it can read, not in a file its code-execution or bash tool can open, not in a fixture its container mounts. Keep them in the grader only. This has to be structural; "don't look at the answers" in a prompt is not a defense, and an agent under optimization pressure will eventually find a `cat evals.json` that wins the game without playing it. If the user's existing eval stores prompts and answers in the same file, split it before the first round. And if the eval derives from a public benchmark and the agent has web or network tools, the answers are also reachable *through the internet* - public mirrors of the benchmark often include solutions. Screen every passing transcript for fetches of the benchmark's repo or solution mirrors (and consider blocking egress to them); treat a pass accompanied by a solution-hosting fetch as invalid, and don't assume any automated contamination check will catch it.

**Handling non-text payloads in traces.** Don't embed raw base64 or binary blobs in the trace text - they bloat the file and render as noise. Instead, write each image or binary payload to a sidecar file and put a markdown reference at the point in the turn where it appeared - `![](<path from flow root>)` for an image, a link for anything else - so the report shows a thumbnail and the analyzer can still open the real file. Only fall back to a bare placeholder like `<[binary: 12kB]>` when the payload genuinely isn't worth keeping.

**Baseline.** Run the unmodified app across **the full set** at the chosen rep count, and record it in `baseline/` and in `_state.json`. Record an environment fingerprint with it - repo commit, lockfile hash, local patches or stubs, disabled tools, model id - and re-verify the fingerprint before each round's run: a changed fingerprint means re-baseline, because the comparison is broken either way. If time or an environment change later separates the baseline from a round, score a no-change control alongside the candidate and gate against the control, not the stale baseline number - identical-code drift of a few points between measurement days is common; where a stochastic build sits between the lever and the score, the control must re-run the build, not just the scorer. The control re-run also estimates the same-environment flip rate for free; don't set a keep gate inside that noise floor (Step 2's threshold sizing). (If cost matters and the pinned reps/split push the total above the Step 2 ceiling, mention it before this run.) You can run the report builder (full or lite - Step 4 has the invocation) on the flow directory now if you want a look: with only a single variant on disk (just `baseline/`, or no `_state.json` yet) the full viewer shows only the Evals and Transcripts tabs - the Summary tab auto-hides until there's a second variant to compare.

---

## Step 4: The loop

Each round is: **analyze -> apply -> run -> record**. Two rules are non-negotiable throughout: only the train split's transcripts are ever read, and neither the eval set nor the budget changes without going back to the user.

**Analyze.** If the next change is already clear - the user named a specific fix, or the last round's result points straight at one - skip the analyzer and write `vN/change.md` directly. Otherwise spawn **one fresh analyzer subagent** (the Claude Code Task tool) and give it: the previous round's `traces/` directory (train-split transcripts only), the **train rows of `results.jsonl`** so it sees every metric's per-case score, the **train rows of `trajectory/scores.tsv`** so it sees how each case has moved across every prior round, the current version of the artifact being iterated on, the scope and off-limits list from Step 1, and **which metric this round is targeting** - on round 1 that's the Step 1 goal, along with the guardrail metrics it must hold. The analyzer scans the scores to pick which transcripts to read, reads however it likes - sort by the target metric, diff low vs high scorers, correlate across metrics, spot cases stuck flat across rounds - and returns a proposed change with a short rationale that cites the specific traces motivating it. Save that rationale to `vN/change.md`. A fresh subagent each round keeps the outer session from accumulating transcript content in its own context.

The outer session **does not read transcripts itself** - for choosing changes it sees scores only. That separation is the data-isolation guarantee: nothing from the held-out test set can leak into a proposed change, because the thing proposing changes never sees anything but train.

Two steers to pass the analyzer. First, **generalize, don't memorize**: the change should describe the failure *behavior*, not the failure *content* - pasting specific nouns or phrases from train cases into the prompt is the fastest route to an overfit change that helps train and does nothing held-out. Second, transcripts show what a block makes the model *do*, not what it prevents or enables without visible action - when proposing to remove something, name which cases you expect to regress, not just which recover. Third, it's fine - especially early on - for the analyzer to propose trying a different lever entirely (a tool description, an API parameter) rather than another wording of the same sentence; exploring where the headroom is can be worth a round.

A sketch of the analyzer's prompt:

> Here is the current `<artifact>` we are iterating on. Below are the train-split rows from `results.jsonl` (each case's full `grade` dict) and the corresponding transcripts. **This round's target is `<metric>` (`<higher|lower>` is better); `<guardrail metrics>` must not regress.** The off-limits list is: `<...>`. Find the cases doing worst on `<metric>`, plus a couple doing best for contrast; name the single behavior that most often costs it, and propose **one** concrete change to the artifact as a unified diff. For every trace you cite as evidence, **quote the relevant lines verbatim** and append its trace file path (`<vN>/traces/<case_id>_rep<k>.json`) - and, if `report.html` was built by the full viewer, the deep link `report.html#tab=transcript&ex=<case_id>&cmp=<vN>&rrep=<k>` - so the user can open that exact rep with one click. Do not reference test cases.

When reading `trajectory/scores.tsv` to pick focus cases, filter to **train rows only** - feeding test-row movement back into the change proposal is a leak of the held-out signal.

**Apply.** (If `approve_each_round` is set, show the diff and one-line rationale and wait for a yes before running; a no counts as a reverted round with the user's reason recorded in `change.md`.) Before applying, do a **de-fluff pass** on the proposed change: cut anything that's a platitude, a restatement of default behavior, or advice with no operational content ("be careful," "think step by step"). Do this every round - fluff accumulates one reasonable-sounding sentence at a time. Then edit the actual files in the user's codebase and save the diff to `vN/change.patch` so it can be reverted cleanly (for a brand-new artifact with no prior version, diff against `/dev/null`). That patch is the round's record of what changed (and what the full viewer's diff drawer renders - click a variant row in Summary), so cut it against the user's real source paths - not a scratch copy under `.claude/hillclimb/` - so each hunk reads as an edit they can apply directly to their repo; if the loop is iterating on a temporary copy, diff the original file instead. Also snapshot the full post-change artifact - the system prompt, skill file, or whatever you're iterating on - to `vN/` (e.g., `vN/skill.md`) so each round is inspectable on its own without replaying patches. If the patch touches a path in `_state.json.harness_paths`, the next run will stop for `--approve-harness`: show the user the diff and wait for their OK rather than running the round unattended.

**Run.** Run the full eval - every case, at the chosen rep count. Running the entire set every round keeps every score in the history directly comparable. Keep the runner's concurrency maxed out (up to the rate limit) - the loop's cadence is gated on how fast each pass finishes - provided retries back off with jitter (Step 0.5); in a shared-quota environment, maxed-out concurrency with a hot retry loop converts someone else's burst into your zero-scores. Write per-case results to `vN/results.jsonl`, the aggregate to `vN/summary.json`, and the **train** transcripts to `vN/traces/<id>_rep<k>.json`. (`trajectory/scores.tsv` is regenerated by the report builder - full or lite - from those files; the runner doesn't write it.)

Two one-case gates before a round's full pass - each costs at most one case, against a pass that costs all of them:

- **Premise-probe config levers.** When the round's change is a config-surface lever (a model parameter, a tool config, `effort` - anything validated server-side at create or call time) rather than prompt content, run one case first and confirm the lever is accepted - and, where the response exposes it, echoed back. A loud validation error is the cheap outcome; the expensive one is a runner that degrades the config error into retries or scored zeros and runs the full set anyway.
- **Print the resolved scope.** Before the pass, have the runner print what it actually resolved - case count × reps × model × estimated cost - and compare it to the approved plan. A dry run that resolves a different case count than the plan is a stop, not a warning.
- **Canary before an unattended or expensive pass** - run one case and compare its error and latency profile to baseline's; if degraded, hold rather than burn the round.

Infra health per round - retries, timeouts, served-model mismatches - is in `errors.jsonl`; a round whose error profile differs grossly from baseline's is void-and-rerun, not a comparable data point.

**Record and report.** Update `_state.json` (round number, and `best` if this round's test score beats it). A number you'll report - including privately-authored held-out cases and their raw results - must land in the recorded `vN/` layout, never only in a run log. If the session or machine is ephemeral (a CI runner, a remote session), also copy each round's `vN/` to storage that survives it. Don't report a round's score until every case has landed - partial reads can show a sign that flips when the batch finishes; if you must report mid-run, flag it as `N/total`. After every round, write the status table (row layout below) at the top of `narrative.md` and keep it current - it is the per-round record, on disk even when the run is headless - then report in chat a one-line headline ("v3 test 0.71 -> 0.74, change: <one line>"). With the full viewer, follow the headline with a pointer to `report.html#tab=summary` instead of the table - the Summary tab is exactly this table, sortable and with the diff drawer one click away. With the lite report, which shows only the primary metric, paste the markdown table under the headline, since it is what carries the guardrail, perf, `$/run` and `spend` columns. `$/run` and `spend` come from `cost_usd` (Step 3: derived by the full viewer when it is on disk, otherwise computed by you from each row's `model` × `usage`). If tracking spend against a budget, cumulative spend is `sum(cost_usd)` over every `vN/results.jsonl`, plus the billed `usage` recorded on failed attempts in each variant's `errors.jsonl` sidecar (failed spend is still spend) - never a maintained counter. The row layout:

> | round | change (one line)    | test  | train | s/turn | out toks      | tool calls   | $/run          | spend  |
> |-------|----------------------|-------|-------|--------|---------------|--------------|----------------|--------|
> | 0     | baseline             | 0.62  | 0.60  | 19.8   | 480           | 3.1          | $2.60          |  2.60  |
> | 1     | when-to-search rule  | 0.70  | 0.73  | 19.5   | 492 (1.0×)    | 4.2 (1.4×) (up) | $2.97 (1.1×)   |  9.70  |
> | 2     | effort=medium        | 0.68  | 0.71  | 11.2   | 310 (0.6×) (down)  | 3.0 (1.0×)   | $1.40 (0.5×) (down) | 12.90  |
>
> Best so far: round 1 (test 0.70). Next: combine round-1 rule with `effort=medium` and re-check latency.

After each round is scored, also **rewrite `narrative.md`** - a model-authored running exec summary of every harness change and its effect so far, not just this round's. Read all of the `vN/change.md` rationales and `vN/summary.json` results and write one short paragraph that says which variant is currently winning and *why*, in terms of the changes ("v4 has the best recall, but v3's prompt tightening traded a little recall for precision and nets the higher overall score"). Overwrite it wholesale each round - it's a snapshot of the story so far, not an append-only log. Keep the current status table above the paragraph. The full viewer's NARRATIVE panel renders this file, and without the full viewer `narrative.md` is the file to open - either way, keeping it current means the user (or a resumed session) can read the state of play at any point in one screen.

Alongside the status table, regenerate the **HTML report** so the user can drill in visually: run the report builder on `.claude/hillclimb/<flow>/` - `shared/evals/report/build-report.mjs` when it is on disk (EAP install), else `shared/evals/report/build-report-lite.mjs` (both paths relative to this skill's base directory), with `node` or `bun` (the selection line and the no-runtime fallback are in build-eval.md §Report builder). The full viewer writes a self-contained `report.html` into the flow directory (exec-summary table and trend charts, side-by-side transcript comparison, and the per-round diffs); the lite builder writes a summary table plus per-case rows with the primary metric per round and links to each trace file. Both write `trajectory/scores.tsv`. It reads the same `summary.json` / `results.jsonl` / `traces/` / `change.*` files you just wrote, so there is nothing extra to produce - run it **after the round's runner has exited**, not while it's still appending (the partial variant's row would show scores from however many cases have landed so far - the full viewer badges it as a partial `N/M cases` run, the lite report just shows the lower case count - don't publish either). **The report is there for when the user wants it, not news to deliver:** give its path once after the baseline and again in the final summary, end each round's status message with the path as a bare last line, and otherwise don't bring it up - no remarks on its size, its notes, or that they should open it - unless the build exits non-zero or they ask. **Re-apply the Step 0.5 spot-checks to the new round's row - in the status table and, with the full viewer, the Summary tab - before pointing the user at it**: every metric and perf column present, plausible, and consistent with this round's change - a `$0.00` cost, `0.0s` latency (or the `usage` / `latency_s` fields behind them missing from the rows), or a column that didn't move the way the change predicts is a runner bug, not a result. Also spot-check that one transcript renders as distinct turn cards rather than a single text blob (full viewer) or that a linked trace file is a JSON list of `{role, content}` turns (lite). The rest are full-viewer features: `build-report.mjs <flow> --check` flags the common trace-format mistakes without rendering; `build-report.mjs --index .claude/hillclimb/` writes an `index.html` linking every child flow's report when several flows run in parallel; and a run directory of a different shape takes a small adapter per `shared/evals/report/SCHEMA.md` - typically a few dozen lines: assemble `Turn[]` from your raw content blocks, compute `RepResult.perf` from `usage`, declare your metrics - handed to `render()` directly.

**Every metric that informs your recommendation must be on the record - on the rows, in the status table, in the report - before you make the call.** If, mid-loop, you compute a new metric in a scratch script and it changes which variant you'd pick, stop and fold it in: add it to the runner's grader so every row carries it, declare it in `_state.json` under `metrics`, re-grade the existing variants in place so the comparison is apples-to-apples, and rebuild the report and the status table - *then* recommend. The user must be able to verify every number behind your recommendation from the flow directory alone (`report.html`, `narrative.md`, the `vN/` files), without your chat history. For metrics that are inherently aggregates - inter-rep consistency, or anything else computed across cases rather than per-row - the same rule holds: put a per-variant comparison table in `metrics.md` so the record carries the comparison (the full viewer renders it), not just your prose description of it.

If the grader is a model-as-judge and the score jumps by more than the change could plausibly explain, **treat it as suspicious before treating it as good news**: spot-check a handful of outputs by hand, show the user, and confirm the judge isn't rewarding a surface pattern the change happened to introduce. Record the concern as a one-line `suspicious` string in that round's `summary.json` so the concern stays attached to the variant (the full viewer shows it as a warning badge). An LLM judge being gamed looks exactly like a breakthrough until you check.

**Separate "did the mechanism engage" from "did it help."** Track a leading indicator of the mechanism firing - how often the agent wrote to memory, called the tool, produced the artifact - as its own column, distinct from the score. It's the in-loop counterpart to the Step 0.5 wiring probe: the probe proved the mechanism *can* work; this proves it *did* this round. A score that moved while the engagement rate didn't is probably noise or a harness artifact - find out which before stacking another change on top.

If train went up and test didn't, the change overfit to the cases the analyzer read - revert it and try a different angle next round. If a round regresses on train too, revert before the next round rather than stacking changes on top of it. If train went *down* but test went *up*, treat it as noise at low rep counts - keep the change only if the pattern repeats on a second run. On ties, prefer the later round.

**Decide.** Check the stopping condition from Step 2. Treat the budget as a **guide, not a wall**: as you approach it with the score still climbing or an obvious idea untried, say so and offer to extend rather than stopping cold. Conversely, if several consecutive rounds haven't moved the score *outside noise* - point estimates drifting but intervals overlapping - you're likely at a plateau even though the numbers look like they're climbing: run Step 4.5's categorization before another content round. To tighten a variant's interval, append more reps to its `results.jsonl` (and baseline's, for a fair comparison) and rebuild - the report recomputes from whatever rows are there; no new round directory needed. Otherwise, once the Step 1 goal has plateaued, offer to change the target before stopping: pick the guardrail metric or perf field with the most headroom that hasn't been tried, and run another round with the analyzer pointed at it under the constraint of not regressing what's already won - but ask first, since the user set the goal and may consider it done. Only stop when no metric has obvious room, or report best-so-far and ask. If the user asked for check-ins and you've hit the interval, report and wait. Otherwise, loop.

---

## Step 4.5: When the loop stalls, categorize before grinding

The analyze -> apply -> run loop assumes each failure is caused by the artifact you're tuning. Once the easy content gaps are filled, that stops being true - remaining failures increasingly come from the grader, the harness, the artifact's structure, or plain variance, and another content round can't move them. The tell is **two or three consecutive rounds where the test score hasn't cleared the noise band** despite changes that should have helped. When that happens, stop iterating content and spend one round categorizing instead.

Spawn a fresh analyzer subagent (same isolation as Step 4's Analyze) to bucket every remaining train-split failure by root cause, reading each transcript far enough to tell which:

| Bucket | Tell | What to do instead of another content round |
|---|---|---|
| **Artifact gap** | Model never had the fact it needed; transcript shows it guessing or searching | This is the loop's home turf - keep going |
| **Grader disagreement** | Model's output looks correct to you but the grader marks it wrong; or the prompt and the rubric ask for different things | Fix the grader, then re-grade *every* variant in place from stored outputs. Before overwriting, compare old vs new grades - how many cases moved, and did the variant ranking change? If the previous best is still the best and its lead over baseline held, keep going. If the ranking flipped or the lead collapsed to noise, the prior rounds were tuned to the wrong signal: show the before/after table and propose restarting the loop from baseline. |
| **Harness / infra** | Case errored before the model produced a scorable output - auth failure, timeout, rate-limit, env setup. Some harnesses *score* the failure instead of erroring it: zero-scored cases whose transcripts carry infra markers (retries exhausted, stall ceilings, empty outputs) belong here too | Fix the harness; exclude errored cases from the denominator until then. For scored-in zeros, decide the handling rule before comparing scores |
| **Structural** | The content exists in the artifact but the model didn't reach it; or the same review finding recurs across rounds; or one dimension (a language, a provider) underperforms regardless of which feature you target | Reorganize - consolidate duplicated facts into one table, split a monolith file, fix the routing - rather than adding more of the unreached content |
| **Variance** | Pass<->fail flips between identical-code runs are as large as the round-over-round delta | You're at the noise floor on this lever. Report best-so-far; offer to raise reps or change target |

A failure that fits none of these is itself a signal: the artifact you're tuning may not be the bottleneck for that slice - offer to change target rather than forcing it into a bucket.

Write the bucket counts to `vN/change.md` in place of a content diff for that round, and tell the user: N of the remaining M failures aren't artifact gaps - here's what each cluster needs. Then dispatch per bucket rather than running another content round against all of them.

Two patterns this surfaces that the per-round analyzer can't:

- **The long tail.** The analyzer's "single behavior that most often costs the grade" is worst-bucket-first and never reaches a tail of many small buckets each costing one or two cases. If categorization shows a dozen dimensions each contributing <=2 failures and none of them have artifact coverage, a one-shot **breadth pass** - draft minimal coverage for every uncovered dimension in parallel, apply all at once - covers more ground in one round than the serial loop will in ten. This deliberately breaks one-change-per-round: the dimensions are independent, the question is coverage not attribution, and no single one would move the score enough to measure on its own.
- **Mid-run grader drift.** Step 0.5 proved the eval was trustworthy at the start. A rubric that's subtly wrong for one feature, or a canonical answer that's gone stale, won't show up as an implausible jump - it shows up as a feature that won't move no matter what content you add. When one bucket resists three rounds of content that looks correct to you, re-read its rubric before writing round four - and if you do change it, re-grade everything, quantify the shift, and decide with the user whether the existing rounds still stand.

---

## Step 5: Report and hand back

When the loop ends, put the codebase at the version that won on test. The headline is the **test-score delta, baseline vs winner** - you already have both numbers from the per-round runs. (If Step 3 chose no split, report the whole-set delta and label it **directional**; if it chose the optional three-way split, run the held-back test slice now on baseline and winner only.) Rewrite `narrative.md` one last time as the final status table (kept on top, as in Step 4) followed by the four-part exec summary - **Recommended change**, **Versus baseline**, **Why trust this**, **What else was tried** - then regenerate `report.html` one last time (full or lite builder, as in Step 4); it is the artifact you point the user at for per-case detail. In parallel, produce a short text report (in the user's PR description if they want a PR, or as a markdown file otherwise) - this is a **companion** to `report.html`, not a replacement, so point at it for transcripts (full viewer) or trace files (lite) and per-case detail rather than duplicating them inline. It covers:

- **Headline:** test score at baseline -> test score at the winning round, each with a confidence interval, and the delta. This is the result. Train improvement is supporting detail, not the claim. **If the test delta is within noise of zero - the CIs overlap, or a paired test over cases isn't significant - say so plainly and recommend not merging.** An honest "this didn't move the needle, here's what I'd try with more budget" is more useful to the user than a dressed-up marginal gain.
- **Per-round table:** round, one-line description of the change, train score, test score, and the guardrail columns with their baseline ratios. Flag only the deltas that clear noise; suppress or grey out the ones that don't, so the user's eye lands on what actually moved. The train vs test columns side by side are the generalization-gap trajectory - if they diverge round over round, say so explicitly.
- The changes that are actually applied to the codebase right now, each with its one-sentence "why" from `change.md`, and each tagged **`[REQUIRED]`** (fixes something broken - e.g., a parameter that errors on the target model) or **`[TUNE]`** (a judgment call that improved the score but that the user could reasonably decline). This lets the user accept the diff selectively.
- **A failure taxonomy, when zeros have mixed causes:** how many failures were refusals, harness or serving errors, and timeouts, versus genuine capability misses - a single rate hides it. And when the loop compared models, classify failures per model before quoting a gap: a failure mode only one model triggers (say, a tool-calling convention the harness rejects from that model) is a harness bug confounding the comparison, not a capability difference; report the gap with and without those attempts.
- **A second-model check, if the artifact will serve more than one model:** re-run the winner once on the other model(s) before recommending it, and report per-model numbers - failures are model-dependent, and a win measured on one model doesn't transfer by default.
- **Two or three before/after transcript pairs** - the same prompt under baseline and under the winning version, side by side - so the user can see the quality change with their own eyes rather than taking the number on faith. Pick cases that illustrate the behavior the changes were targeting.
- **What you'd try next.** Proactively list the concrete levers still on the table - "`effort=medium` looked promising on latency but I didn't re-tune the prompt for it; the `summarize` tool description is still vague; judge could move to `claude-sonnet-5`" - rather than waiting for the user to ask whether there's more. Include anything that seemed to need a bigger change than the target allowed.

The report is what lets the user trust the diff enough to merge it. Be specific about *why* each change helps; "reworded the system prompt" is not enough.

If the eval and flow directory aren't already committed, offer the same three-way choice as `build-eval.md` §Make it durable (commit eval / commit eval + transcripts / don't), with fresh file and size counts now that there are multiple rounds. Recommend committing the eval - the harness change going into this PR is only half the value; the other half is being able to re-baseline on the next model without rebuilding the eval.

---

## Failure modes to avoid

- **Touching the held-out set.** The split is the only thing standing between a real improvement and an overfit one. Don't open test transcripts, and don't let test-case content inform a proposed change - the analyzer reads train, the headline comes from test, and that wall is the result's credibility.
- **Leaving ground truth reachable.** If the model-under-test can read the expected answers from disk, the loop will eventually find that path and "win" without improving anything. Isolate answers structurally; don't rely on instructions. On a public benchmark, "reachable" includes the internet - a web-enabled agent can fetch a solutions mirror (see Step 3).
- **Trusting an implausible jump.** When a model-graded score leaps further than the change could reasonably explain, the likeliest cause is the judge being gamed, not the app getting better. Spot-check by hand before celebrating.
- **Climbing on an untrustworthy eval.** Skipping Step 0.5 means you might be tuning against a disconnected mechanism or a mis-aggregated headline number - "improving" an artifact and shipping nothing. Prove the mechanism is wired and the headline recomputes from raw per-case results *before* the first round.
- **Grinding content at a wall.** Adding more content for a feature that hasn't moved in three rounds, without first asking whether the failure is the grader's, the harness's, or structural. Step 4.5 is the check.
- **Averaging refusals into the score.** A safety refusal is not a capability failure, and a harness-killed attempt is neither. Record a failure class per attempt (refusal / harness-or-serving error / timeout / genuine failure), report refusal-zeros separately from capability-zeros, and pre-register the scrub predicate for anomalous zeros (e.g. completed with score 0 at a wall-clock and request count far below the task's normal floor), reporting raw and scrubbed - decide the rule before the scores exist.
- **Letting the platform shift under the loop.** If the serving platform or harness changes execution semantics mid-run - environment reuse, batching, defaults - rounds stop being comparable. Pin the runner and its dependencies for the whole loop, and verify the semantics you depend on from run artifacts (e.g. one environment per case) rather than assuming them.
- **Climbing on a gain you can't explain.** If the score moved but you can't point to the behavior that moved it - a shift in how often the mechanism engaged, specific flips in the traces - the "win" is as likely noise or a measurement artifact as a real improvement. Tie every delta to a mechanism; distrust the ones you can't.
- **Overgeneralizing from one case.** Seeing a pattern in a single failure and rewriting the whole prompt around it is the most common way to make the score go down. The analyzer should cite the traces that motivate a change, describe the *behavior* rather than pasting the *content* of the failures into the prompt, and prefer narrow changes over sweeping ones.
- **Untracked bundling.** Stacking several unrelated edits into one round's change means that if it helps you won't know which part did the work, and if it hurts you won't know which part to revert. One idea per round.
- **Accumulating fluff.** Without the per-round de-fluff pass, prompts grow a sediment of vague, unfalsifiable advice that costs tokens and dilutes the instructions that matter.
- **Losing state.** If the session is interrupted, the next session should be able to read `_state.json` and the `vN/` directories and pick up exactly where this one left off. Write state after every round, not at the end. If the proposer/analyzer runs as its own long-lived session, its working artifacts - the traces and scores it was handed, and its rationale for each change - belong under the flow directory too, so an interrupted loop can reconstruct the proposer, not just the scores.
- **Editing off-limits content.** The user told you what not to touch in Step 1. A change that improves the score but violates a constraint the user stated is not an improvement.
- **Treating the budget as a wall (or ignoring it)** - when cost is a guardrail. Recompute cumulative spend from the `results.jsonl` files - plus the billed usage on any `errors.jsonl` failed attempts - and check it against the Step 2 ceiling every round. If you're going to exceed it, ask first. Equally, don't stop cold at the limit while test is clearly still climbing without *offering* to continue.