Building Wheels apps with AI coding agents

Our aim was a like-for-like comparison with the way Ruby on Rails' own ai-evals measures agents: the same harness, the same agent, a comparable sample app, and the same style of task with hidden acceptance checks. Where we couldn't match Rails exactly, we say so.

1. What we measured, and why

• Same harness and agent as Rails' ai-evals. We used Rails' lemans harness and its miniswen agent unchanged, so a Wheels task is run and scored the same way a Rails task is.
• A comparable sample app. Tasks run against a Writebook-style content app built in Wheels — the same kind of app Rails' Stage 1 uses — so the asks line up.
• Frozen tasks with hidden checks. Each task is a plain-English feature request. The agent gets the prompt and nothing else; a hidden spec decides pass/fail. A task scores 1 only if the app's own test suite is green and every hidden check passes. Tasks and their checks were frozen and independently reviewed before any model ran. The only changes after any model ran were the two over-strict check corrections disclosed in section 5, each reported with both the before and after figures.
• Two arms. Each model ran twice where possible: bare (a stock Wheels app) and shipped docs (the same app with the AI context Wheels now ships — .ai/ docs and a prebuilt .mcp.json). This isolates whether the shipped context helps.
• Agent-native rows reported separately. Alongside the like-for-like rows we ran Claude Code against the same frozen tasks, via the "Harbor" setup. Those rows are a different agent and are always reported and labelled separately — never pooled with the like-for-like numbers.

Ten tasks (plus a hello-world control) × three attempts per cell. A "cell" is one model in one arm. Twenty-four cells in all: nine models in both arms, four more in the shipped arm only, and two Claude Code cells.

Integrity first. The case we wanted to test honestly is simply whether Wheels deserves a seat at the table. So: no benchmark-specific caching, no pre-generated answers, and no app state an agent wouldn't have in normal use. Every improvement we made to help agents shipped to every Wheels user — in wheels new, the docs, and the MCP server — not just to the test. Every run was recorded, including failures and infrastructure errors, and those are reported below.

2. Latest results (October 2026)

Overall, across the ten tasks, the agents passed 700 of 790 scored trials (88.6%) — raw, before two hidden-check corrections (section 5), 660 of 790 (83.5%).

Steps are the agent's tool-use turns (a guard stopped a trial at 150). Spend is the settled provider cost for that cell. The "docs" arm is the shipped AI context; "bare" is a stock app.

Notes:

• Pass counts scored trials, with the two check corrections of section 5 applied; the raw figure is in parentheses where it differs.
• Qwen shows /32: one trial in each arm was refused by the provider's content moderation and is out of the denominator (section 5).
• Claude Code rows can't report a step count the same way (a different agent). Both lost three trials of one task to a scoring artifact that couldn't be re-scored for that agent (section 5); on the like-for-like pattern each cell would be 32/33.
• Spend was well within budget on every cell; no single trial reached the per-trial cost limit (the most expensive single trial was $12.27, under the $20 cap).

3. Did the shipped AI context help?

For the nine models that ran in both arms, with both check corrections applied:

Honestly: the effect is within noise. After correcting the two over-strict hidden checks (section 5), the shipped docs did not produce a measurable across-the-board lift — the pooled difference is a single trial, in the noise at three attempts per cell. Most of the apparent raw gain (+8 trials) came from one of the corrected checks, which the docs arm happened to pass more often by chance.

Where the docs did help was narrower and more plausible: a model occasionally used a 4.2 API it would otherwise have missed, and a couple of models took fewer steps with the docs present. Cost and step counts barely moved with docs either way (within about ±10% on spend, under one step for most models).

We read this as: the strong models already know how to build a Wheels app from the stock conventions, and the shipped context mostly helps at the margins and on newer APIs. We'd rather report that plainly than overstate it.

The bare/docs totals here were recomputed from the rescored data plus the erase-account per-trial flips, and confirmed by re-running the tally: bare 256/296, docs 255/296 (raw 236 vs 244).

4. The ten tasks

Seven tasks mirror a Rails ai-evals task; three are Wheels-native (no Rails counterpart). Pass rate is across all 24 cells, rescored.

(hello-world control: 72/72.)

What failed, and why:

• ar-erase-account asks for an atomic, hard delete of an account and everything only it owns, through a soft-delete model where the default delete hides rows rather than removing them. Most of the genuine misses were real model mistakes: a soft-delete-only erase that leaves rows behind, calling object methods on raw query rows, a missing datasource, and one route-constraint escaping bug (below). Two more were a documentation gap, not a model error — a raw SQL subquery with a literal inside where= threw an unhelpful error, since addressed (#4531). Two weaker models (GPT-5.6 Luna, Haiku 4.5) account for most of them.
• hw-scoped-broadcast asks the agent to add per-listener authorization to a live-update stream that originally had none. Excluding the two weakest models, it passes 93%; the rest is almost entirely those two.
• wn-reversible-trash and wn-literal-route are Wheels-native tasks about soft deletes and route declaration order; the misses are again concentrated in the two weakest models.
• Haiku 4.5 is the clear floor (12/33 bare, 9/33 docs) — its failures are spread across most tasks, not one weakness.

A recurring, framework-agnostic lesson showed up too: CFML strings have no backslash escapes, so a digit route constraint must be written "\d+", not "\\d+" — one agent got this wrong and broke the app's reading URLs. (We've since made that explicit in the routing guide.)

5. Corrections and deviations, in full

We'd rather show the rough edges than hide them.

• Two hidden checks were over-strict and were corrected after the runs. Both figures are kept; the corrected ("rescored") one is primary.
  1. wn-literal-route demanded every section's title in the print view, when the task asks for each section's content (and the app's own view renders a section by its text). Re-scored with that one assertion relaxed and everything else unchanged, the task went from 27/72 to 60/72. Of the twelve trials that don't reach that figure, six are genuine failures (an HTTP 301 on the print route) and six are Claude Code trials that couldn't be re-scored (see below).
  2. ar-erase-account treated a book as "edited by nobody else," but a fixture user was an administrator and the app lets administrators edit every book — so "nobody else can edit" was ambiguous against the app's own rule. Re-scored with that fixture user demoted (every other check unchanged), the task went from 53/72 to 60/72; seven trials that kept the book because an administrator exists now pass, and no passing trial became a failure. An eighth trial of the same kind — a Claude Code trial — couldn't be re-scored because its diff wasn't captured, so it stays as originally scored.
  Each correction changed exactly one thing; every other check stayed as frozen.
• Provider refusals. A provider's content moderation refused two Qwen3.8-27B trials of one task (and its retry); those two trials are out of the denominator, reported as /32.
• Infrastructure, not the model. A duplicate launch was discarded and re-run; a pricing-metadata bug cost two trials at step one (fixed, re-run); a couple of transient provider 503s were retried. Eight re-runs in all ($4.74). None changed a scored result; the originals are kept as evidence.
• One agent-native web fetch. In one Claude Code trial, the agent fetched a public Wheels upgrade guide page twice and solved the task. The Claude Code rows allow web access by design (it's an agent-native setup), so this is within that row's definition — but it's the first time it actually happened, so we flag it: that one task's Claude Code result isn't strictly comparable to the like-for-like rows.
• Three Claude Code trials per cell couldn't be re-scored for the corrected wn-literal-route check, because that agent's diff wasn't captured. They stay as originally scored.

6. What changed in Wheels because of this

Per our integrity rule, everything the rounds surfaced was fixed for all Wheels users, not just the test. Running real agents at real tasks turned out to be an excellent way to find rough edges. A selection of the user-facing changes the rounds drove, each a public PR:

• savedChanges() and saved-change state after a save (#4349) — so an afterCommit can tell what the commit actually wrote, instead of apps hand-keeping flags and getting the lifecycle wrong.
• transaction="savepoint" for nested transaction units (#4040).
• An application data cache — appCacheFetch() and friends (#4388).
• Query assertions in specs — assertQueries() and friends (#4357).
• A clock seam for tests — travelTo() (#4387) — and a test client that can set the request's client address (#4343).
• Forms inside cached fragments work correctly (#4032); production error pages return the right HTTP status (#4238); date comparisons in the query builder match correctly on SQLite (#4281); a failed background-job enqueue now raises instead of being silently dropped (#4266), and a job enqueued inside a datasource transaction is deferred to its commit so it runs predictably (#4282).
• Dozens of smaller docs, CLI, and upgrade-path fixes — a fresh-VM install-and-upgrade walkthrough, driven by watching agents (and new users) hit the same cliffs, produced a long run of corrections to wheels new, the upgrade commands, and the guides.

Several of these — the application cache, query assertions, the clock seam — are new in 4.2 precisely because an agent needed them and we agreed every Wheels developer would too.

7. How we kept it honest

• Frozen before any model ran. Tasks and their hidden checks were fixed and independently reviewed before the first run. The only post-run changes were the two over-strict check corrections above, which relaxed over-strict checks and are reported with both figures.
• Every run recorded. Passes, failures, refusals, and infrastructure errors are all kept. Infrastructure failures are reported separately from wrong answers, and re-runs are accounted for.
• No benchmark-specific caching or pre-seeded answers. The app is a normal Wheels app; the shipped context is the context every Wheels user gets.
• Same harness, agent, and task style as Rails' ai-evals for the like-for-like rows; the agent-native rows are labelled and reported on their own.

Credits

Models and providers tested: OpenAI (GPT-5.6 Luna, GPT-5.6 Sol, GPT-6 Astra), Anthropic (Opus 5.5, Sonnet 5.5, Haiku 4.5, Fable 5.1), Google (Gemini 3.8 Flash), DeepSeek (V4.1 Flash), Alibaba (Qwen3.8-27B), Moonshot (Kimi K3), Zhipu (GLM 5.3), xAI (Grok 4.7). Harness and agent: Rails' lemans / miniswen (ai-evals) and Anthropic's Claude Code.