I Benchmarked My Claude Code Presets. The Cheap Ones Passed Every Check.

I start Claude Code every day with one of five presets, and until this month I was choosing between them on gut feeling. The choice is not cosmetic: the same request can cost twenty times more on Opus at maximum effort than on Sonnet. So I built a small benchmark, ran all five presets through four real kinds of work, three times each, and graded every run twice: once with a deterministic check, once with an LLM judge.

The short version: the cheap presets passed every check, and the most expensive settings bought almost nothing. This post is mostly about the longer version, though: how the harness works, how the graders were built and tested, and what nearly broke the results along the way. If you want to run the same experiment on your own presets, everything is in the repository.

TL;DR

• 5 presets × 4 task types × 3 runs: 60 runs, 54 valid, 10.8 hours, $230.93 of agent usage plus $14.96 for the judge, Claude Code 2.1.303.
• The light presets won all four task types under a decision rule fixed before the runs: Sonnet as the main model, an Opus advisor, Haiku subagents. Mean cost $0.63–0.75 a run.
• Opus at medium effort (normal) is the best quality per dollar: judge median 9.0 on three task types and 8.8 on the fourth, at $1.70 a run.
• Opus at xhigh with a Fable advisor, and Opus at max, never beat normal's judge median. They cost $8.84 and $9.34 a run.
• Most of the engineering went into the graders, not the harness. The greenfield task's grader has 20 hidden tests and a self-test with 56 variants, and it was reviewed by Claude Code and Codex independently.

// Why I Built It

One Claude Code setup for everything didn't work for me. Claude Code lets you choose the main model, the effort level (how hard it thinks), an advisor model it can consult on hard calls, and the model its subagents run on. Opus at maximum effort burned through my plan's limits and was sometimes simply too slow; a one-line fix doesn't need it. So I made a ladder of presets, from Sonnet with Haiku subagents for routine work up to Opus with a Fable advisor for hard problems, each a one-word oclaude-* command.

Then, on 8 October 2026, a post on X presented as an Anthropic tip recommended "Sonnet builds, Haiku explores, Opus reviews": let Sonnet do the work, send exploration to cheap Haiku subagents, keep Opus as an advisor for the hard calls. Close to Opus quality for a fraction of the spend. It was backed by anecdotes, and its one-line command used a --subagents flag that Claude Code doesn't have (unknown option on 2.1.295; the author later corrected it). The real mechanism is the CLAUDE_CODE_SUBAGENT_MODEL environment variable, which my light preset already used.

If the recipe holds, I was overpaying on most of my work. If it doesn't, the cheap preset was quietly costing me quality. Before changing a daily habit I wanted numbers, so the question became concrete: for the kinds of work a developer does every day, which preset gives the best result per dollar and per minute, and does the recipe hold up against plain Opus and against Opus with an advisor?

// The Five Presets

Preset	Main model	Effort	Advisor	Subagents
high	Opus 5.5	xhigh	Fable	Sonnet
normal-max	Opus 5.5	max	—	default
normal	Opus 5.5	medium	—	default
light	Sonnet 5.5	medium	Opus	Haiku (default)
light-force	Sonnet 5.5	medium	Opus	Haiku (forced)

The arms are designed in pairs that differ in one factor: normal and normal-max differ only in effort, light and light-force only in forcing Haiku. light is the recipe as published. high is the hard-task preset rather than an isolating pair.

Each preset is a shell script that execs claude with its flags and passes through whatever the caller appends:

#!/bin/bash
# high: Opus main model, Fable advisor, Sonnet subagents, effort xhigh
unset CLAUDE_CODE_SUBAGENT_MODEL_FORCE
export CLAUDE_CODE_SUBAGENT_MODEL=sonnet
exec claude --dangerously-skip-permissions --effort xhigh --model opus --advisor fable "$@"

Run with no arguments, it is an interactive session. install-launchers.sh symlinks every config onto PATH as oclaude-<name>, so the benchmark arms and my daily launchers are the same files. That was the point: I wanted to measure what I actually use, not a lab version of it. Two rules follow from it. Every config pins --model and --effort explicitly, because a config that inherits the settings default drifts the day the default changes. And every config without FORCE unsets CLAUDE_CODE_SUBAGENT_MODEL_FORCE, so a value exported in my shell can't leak into an arm.

Why there are two light presets

CLAUDE_CODE_SUBAGENT_MODEL turned out to be a default, not an override:

• A subagent whose frontmatter sets model: keeps its own model. The pr-review-toolkit agents, for example, are pinned to Opus.
• Agents declared model: inherit, which includes the built-in Explore and Plan agents, run on the main model. So in light, Explore runs on Sonnet, not Haiku.
• CLAUDE_CODE_SUBAGENT_MODEL_FORCE (any non-empty value) puts every subagent on that model, inherit included. Only forks and skills with model: inherit keep the main model.

The recipe as posted therefore doesn't do what it says for exploration. light-force is the version that does, and the pair shows whether forcing matters.

// The Harness

bench.py is one file of standard-library Python 3.8+, with four commands: list, smoke, run and summarize. A run of one preset on one task goes like this:

1. Fresh workspace. The task's workspace/ folder is copied to a new temp directory and committed as base, or, for a task on a real repository, a detached git worktree is created at a fixed commit.
2. Optional setup.sh runs inside it.
3. The preset runs headless, with the harness appending -p <prompt> --output-format json --no-session-persistence --max-turns N --max-budget-usd X.
4. Metrics come from Claude Code's JSON result: wall time, API time, turns, tokens, and modelUsage, which gives the cost per model. That makes the advisor's share of the bill visible, along with subagent counts and permission denials.
5. The diff of the workspace against base is captured.
6. check.sh, the deterministic grader, runs in the workspace. Exit 0 is a pass; a number between 0 and 1 on the last line is a partial score.
7. The judge, a separate Claude call with no tools at all (--tools ""), reads the rubric, the prompt, the agent's final message, the diff and the check's verdict, and returns a 0–10 score as structured output.
8. One row is appended to results/runs.csv, which is append-only. Raw JSON, final messages, diffs and judge output go to a gitignored results/raw/.

Runs execute one at a time, in interleaved order (A, B, C, A, B, C), so an afternoon of slow API latency or a warm prompt cache doesn't land on one preset. The harness has its own unittest suite that runs against a fake claude binary, so testing the harness never calls a model. CI runs it on Python 3.8 and 3.14, plus pylint.

Three things I learned about Claude Code while building it:

• --advisor and --max-turns work in -p mode even though claude --help doesn't list them.
• total_cost_usd is Claude Code's client-side estimate from its bundled price table. On a subscription seat it is a proxy for plan usage, not an invoice, but it is consistent across arms, which is what a comparison needs.
• On a small task, the session context dominates the bill. In the first calibration pass on a trivial bug fix, 72–85% of each run's cost was writing the 32–35k-token session context to the one-hour prompt cache, at twice the input price. That is why Opus at max cost only 1.3× Opus at medium on that task, not the roughly 5× that Claude Code's own effort cost index suggests.

// Four Task Types

A recipe that shines on one kind of work can fail on another, so the four tasks have deliberately different shapes:

Task	Shape	Grading
Fix a lazygit bug: honour git's conflict-marker-size attribute, starting from the commit before the upstream fix (PR #5902)	Exploration in a large real Go codebase	In a container without network: 8 hidden integration tests, behind gates on the build and lazygit's other 512 integration tests
Review a pull request: a 600-line PR to linkbox, a small Go service written for the benchmark, using the pr-review-toolkit plugin	The daily team workflow	10 planted problems and 5 decoys that look wrong but are correct; score = (found − decoys flagged) / 10, pass from 0.60
Write an architecture decision record: how linkbox should store its links, from a brief, seven team notes and the code	A document, no code	Gated on the file and its sections; the judge scores ten grounded points. Only one store fits every constraint, and the brief leans towards a wrong one
Build meowdoku: a 10×10 logic-puzzle web game in Go with htmx 4	Greenfield, with a library newer than the models' training data	20 hidden acceptance tests behind gates: offline build, gofmt, vet, the agent's own tests

The judge is Opus at high effort. For the review and the decision record, the rubric carries the answer key, because the judge can't see the materials the agent read.

The decision rule was fixed before any real run. Rank presets by checks passed, keep those within one judge point of the best, and take the cheapest. A preset wins outright only if it is at least 25% cheaper than the next candidate; closer than that is a tie.

// Keeping It Fair

Most of the effort went into making sure the runs measured the presets and nothing else.

• A clean room for every run. Each run gets a fresh copy of its workspace. The harness strips the environment variables that would silently override a preset (CLAUDE_CODE_EFFORT_LEVEL, CLAUDE_CODE_SUBAGENT_MODEL, CLAUDE_CODE_SUBAGENT_MODEL_FORCE), and, when it is started from inside a Claude Code session, that session's own variables (CLAUDECODE, CLAUDE_CODE_CHILD_SESSION, its messaging socket and token), which would otherwise make every run a child of my session. It logs only their names, because one of them is a token.
• None of my own tools. Calibration showed that runs inherited my MCP servers, and agents that wanted to check their work drove my real Chrome. Runs now see exactly one MCP server, a headless browser in a throwaway profile, plus only the plugins their task declares, and no claude.ai plugin sync. The judge gets no tools at all.
• Caps from calibration, not guesswork. Each task ran once on every preset first, to set its turn, budget and time caps with room to spare. A capped run counts as a failure, so a cap set too tight would punish the slow presets for my mistake.
• Graders tested like code. Every check has a self-test that grades known-good and known-bad results, from a perfect answer to an agent that edits code it was told to leave alone. Every change went through automated code review before it was merged.

// Building a Grader You Can Trust: the meowdoku Case

The greenfield task shows best where the time actually went. I'll walk through it in some detail, because the method carries over to any agent benchmark.

The task

Meowdoku is a Queens-style logic puzzle: a 10×10 grid split into 10 coloured regions, one cat per row, column and region, and no two cats touching, diagonals included. The agent builds it as a Go 1.27 web app, standard library only, with server-rendered html/template, htmx 4.0.0 for every server interaction and Alpine.js 3.17.4 for client-only state. The interesting part is the generator: it has to produce puzzles with exactly one solution, deterministically per seed, in under two seconds.

htmx 4 was a deliberate choice. It was released in August 2026, after the models' training data, and npm's latest tag still pointed to 2.0.11. It breaks htmx 2 habits in ways a model working from memory gets wrong: attribute inheritance must be declared explicitly with :inherited, 4xx and 5xx responses are swapped into the page by default, and the events are renamed (htmx:afterSwap is now htmx:after:swap). The prompt says only "htmx 4 is not htmx 2: check the migration guide instead of relying on memory". Getting the rest right is part of the test.

Contract first, tests hidden

A greenfield task can't be graded by diffing against a known fix, so the prompt pins a contract and leaves everything else open. The engine API is fixed:

func Generate(seed int64) Puzzle
func (p Puzzle) Validate() error
func CountSolutions(p Puzzle, limit int) int
func Solve(p Puzzle) (cols [Size]int, ok bool)
func Check(p Puzzle, b Board) (conflicts []Pos, won bool)

So are the HTTP routes and status codes, the state JSON, an HX-Trigger header that fires a won event on the winning move, and the data-* attributes the tests read from the markup. The prompt tells the agent plainly: "Acceptance tests you cannot see exercise exactly this contract. Everything else is your call."

The 20 hidden tests (TestHidden*) were written from the contract before any implementation existed, so they test the contract rather than one particular implementation. They use their own independent solver as the oracle, never the agent's Solve or Check. At grading time, check.sh:

1. copies the workspace to a temp directory, deletes the agent's own tests there and drops in the hidden ones, so there can be no name clashes;
2. runs the engine tests without -race, because they include timing, and the web tests with -race, because they include a concurrency test;
3. runs everything offline: GOPROXY=off GOTOOLCHAIN=local GOWORK=off, with the user's GOFLAGS cleared.

The gates come in two kinds:

• Hard gates set the score to 0: go.mod or the vendored htmx and Alpine files changed, or the offline build fails, which is also how a third-party import gets caught.
• Soft gates fail the run but keep its score: gofmt, vet, the agent's own tests, and a handful of structural checks such as the binary actually serving on -addr and the engine importing no I/O packages.

The score is the fraction of hidden tests passed, so a nearly correct build scores 0.95 rather than 0.

Testing the grader

A grader is code, and untested code is wrong. The self-test builds workspaces exactly the way the harness does, runs check.sh on each, and compares the verdict with what it should be:

• The reference solution must pass at 1.00, and the untouched workspace must fail at 0.00.
• Broken variants must each fail, and in the right named test: Check ignoring diagonal contact, a generator that skips its uniqueness check or uses the global random source, a game store without its mutex, a move accepted after the win, one changed byte in htmx.min.js.
• Correct-but-different variants must pass at 1.00: HX-Trigger sent as JSON instead of a bare name, an absolute Location header, region ids numbered differently, Content-Type without a charset, scripts loaded through <base href>.

The third group is the one most benchmarks skip, and it's the one that catches overfitting. A test that only accepts the reference author's style of correct code silently penalises every other correct agent.

Two independent reviews

The grader was written by Claude Code on the high preset from a spec I drafted in a Claude chat. Before committing, I had it reviewed twice with the same prompt: once by a fresh Claude Code session on a different preset, and once by Codex, a different model family. The review prompt asked each reviewer to trace every assertion back to a sentence in the contract and to flag anything that depended on what the contract leaves open.

Together they produced 29 findings plus four nits, with heavy overlap. The rule for handling them was strict: reproduce every finding as a self-test variant before fixing it. "A correct implementation would fail" became a correct variant that had to fail on the old grader and pass on the fixed one; "a wrong implementation would pass" became a broken variant with the opposite expectation. Where the reviewers disagreed on a fix, the decision came to me, and a finding could be rejected only with evidence. Some of what they found:

• check.sh counted printed --- PASS lines while ignoring the test binary's exit code.
• The won trigger was matched as a substring, so any header containing those letters passed.
• Any element with a data-row attribute counted as a board cell.
• GOFLAGS set in the user's go env file leaked into grading.
• macOS /bin/bash 3.2 left the temp directory behind.
• Go's ServeMux answers HEAD on every GET route, while the prompt said "any other method answers 405". That one was fixed in the prompt, not the test.
• A generator that cycles through 16 fixed puzzles scored 0.95, because nothing in the contract forbade it. The prompt gained a sentence on variety.

In the end the default self-test group held 56 variants, and all of them matched their expected verdicts under both bash 5.3 and macOS /bin/bash 3.2. The remaining known gaps went into the plan as threats to validity rather than being hidden: an agent could in principle fake passes with os.Exit(0) in an init function, determinism is checked on nine seeds only, and the web tests have no process isolation.

Building the grader took longer than building the game. That ratio is the main lesson of the whole project.

// What Went Wrong on the Way

Calibration exposed problems that would have skewed the results.

Caps have to come from the most expensive arm. The first meowdoku calibration pass, one run per preset, all passing every hidden test, looked like this:

Preset	Time	Cost	Turns	Advisor cost
light	557 s	$0.87	32	not consulted
light-force	661 s	$0.82	27	not consulted
normal	704 s	$3.13	69	—
high	2,527 s	$13.88	127	$4.41
normal-max	timed out at 3,600 s	—	—	—

Caps derived from the Sonnet arms would have cut off every Opus run at high effort and counted it as a failure. The caps were set from the expensive arms instead.

A plugin was leaking answers. A security plugin's hook reviewed each session's git diff when the session ended and handed its findings back to the agent. In the review task, that diff was the planted pull request, so the hook could hand any preset three of the ten problems for free. Runs now get only the plugins their task declares.

Decoys punished good reviewers. Three decoys sat on lines where a careful reviewer could make a different, valid point, and lost points for it. I changed the code around the decoys so each one invites only the mistake it tests.

A grader that passed nothing. One fix to the document task's check read git state the wrong way. The self-test passed, but in a real run the check would have failed every answer. Code review caught it, and the self-test now stages the workspace exactly as the harness does.

A broken turn counter. In the lazygit task, normal-max hit the 100-turn cap twice, both times with working code. That cap had been set during calibration, when background subagents were breaking the turn count. So that 0/2 says as much about the cap as about the preset.

The weekly usage limit. Halfway through the experiment my account hit its weekly limit, and six runs were refused or cut off. They are excluded from the results. The harness now records a usage-limit response as its own result type and stops the experiment, instead of recording every following run as a failure.

The judge isn't free. On the trivial calibration task, the five judge calls cost more than the five runs ($1.49 against $1.35), at 50–90k tokens of context per Opus call. On the real tasks the judge came to $14.96 against $230.93 of runs.

// Results

Every valid run passed its check, with one exception: the two normal-max lazygit runs that hit the turn cap. Judge scores sat between 7.5 and 9.0. Each cell shows the median judge score, the median cost per run and the checks passed; bold marks the winner under the decision rule.

Task	high	normal-max	normal	light	light-force
lazygit fix	9.0 · $6.64 · 2/2	8.0 · $9.65 · 0/2 (turn cap)	9.0 · $1.70 · 3/3	9.0 · $0.59 · 2/2	8.5 · $0.70 · 2/2 (tie)
PR review	9.0 · $13.90 · 3/3	8.5 · $12.31 · 3/3	9.0 · $2.65 · 3/3	8.5 · $1.66 · 3/3	9.0 · $0.78 · 3/3
ADR	9.0 · $2.18 · 3/3	9.0 · $1.84 · 3/3	9.0 · $0.37 · 3/3	8.0 · $0.16 · 3/3	7.5 · $0.17 · 3/3
meowdoku	9.0 · $11.83 · 3/3	9.0 · $15.74 · 2/2	8.8 · $2.27 · 2/2	8.0 · $0.63 · 3/3	8.0 · $0.75 · 3/3 (tie)

Averaged over all valid runs:

Preset	Mean cost per run	Judge median	Median time
normal-max	$9.34	9.0	1,111 s
high	$8.84	9.0	618 s
normal	$1.70	9.0	358 s
light	$0.75	8.0	224 s
light-force	$0.63	8.0	209 s

// What the Numbers Say

The recipe mostly works. The light presets passed every check and won all four task types under the decision rule, at less than half the cost of plain Opus and about a twelfth of the Opus presets at high effort.

They gave something up. Against plain Opus, the light presets scored a point to a point and a half lower on the decision record and 0.8 lower on the greenfield build: the tasks where the judge rewards depth and detail. On the repository fix and the review, they matched Opus or came within half a point.

The savings come from Sonnet doing the work, not from Haiku. Sonnet consulted its Opus advisor only during the review task. Forcing every subagent onto Haiku moved scores by half a point at most, and the meowdoku runs spawned no subagents at all, so there was nothing for Haiku to take over.

More effort bought almost nothing. normal-max never beat normal's judge median and was the slowest preset on every task. The Fable advisor accounted for a large share of high's cost ($4.41 of $13.88 in the meowdoku calibration run), and high beat plain Opus only on the greenfield build, by 0.2 points, at about five times the cost.

// What I Use Now

• oclaude-light / oclaude-light-force for routine work: fixes, reviews, documents, small builds. Every check passed, at $0.63–0.75 a run.
• oclaude-normal when the result matters most: Opus at medium effort scored 9.0 on three task types and 8.8 on the fourth, at $1.70 a run.
• oclaude-high and oclaude-normal-max kept for experiments: they beat normal only on the greenfield build, by 0.2 points, at five times the cost.

// What This Can't Tell Me

• Every task ended near its ceiling. All presets solved all tasks, and the judge separated them by a point at most. These results can't show where the expensive presets start to pay off. Finding that point needs a harder task, one where a quick pass gets it wrong.
• The judge is Opus, the same model that runs three of the presets. A judge that favours its own style would favour them, so if anything the light presets look slightly worse than they are. A judge from another model family would be cleaner.
• Small samples. Six of the twenty cells hold two runs rather than three, because of the usage limit.
• A snapshot. The results belong to these models and Claude Code 2.1.303. I plan to rerun the benchmark when either changes, which is exactly what the harness is for.

// Run It on Your Own Presets

You need Python 3.8+, git and the claude CLI. Put your launchers in configs/, each ending in exec claude … "$@", then:

git clone https://github.com/olegiv/claude-code-preset-bench
cd claude-code-preset-bench
python3 bench.py list                    # configs and use cases, no model calls
python3 bench.py run --dry-run --runs 1  # print the exact commands, no model calls
python3 bench.py smoke                   # one cheap call per config: auth, flags, billing
# calibrate caps on the cheapest and the most expensive arm
python3 bench.py run --runs 1 -c light normal-max -u meowdoku
python3 bench.py run --runs 3            # the real experiment (costs real usage)
python3 bench.py summarize               # tables on stdout and results/summary.md

A use case is a folder with prompt.md (frontmatter with caps, then the prompt), a workspace/ or a repo: to start from, a check.sh and a rubric.md. Copy usecases/_template/ to start. Budget before you scale up: on these four tasks, three repeats of five presets came to about $250 including the judge.

The harness, the tasks, the graders and every run are in github.com/olegiv/claude-code-preset-bench. The full experiment log is in PLAN.md, and the numbers in results/summary.md.

The takeaway I didn't expect is not about Sonnet or Opus. It's that a benchmark of AI agents is mostly a benchmark of your graders. The model runs were the cheap, fast part. Making sure a check accepts every correct answer and rejects every wrong one was where the work was, and it's the same engineering that moved upstream everywhere else in my workflow (I wrote about that in April).

— Oleg Ivanchenko, October 2026