A saving only counts if the result holds.
Every measured saving on this site comes with the quality of the same tasks — judged against criteria that neither Claude nor Grace saw.
How quality is measured
Each task has acceptance criteria that are kept secret from both Claude and Grace. Quality is the share of those criteria a result meets, judged per task. The baseline is Claude Opus 5.5 alone with the token-saving plugins caveman and context-mode, on the same tasks.
New builds with Grace 0.3.8 from the tests of October 2 – 3, 2026 (subscription runs, not API mode). Changes to existing code: only an earlier measurement in API mode on September 28, 2026, on an interim build — not the release version. The subscription test of September 29, 2026, an earlier version. The subscription test’s quality level by level, and the raw data per task, are on the measurements page. The tests of October 2 – 3, 2026 — Grace 0.3.8 on 60 tasks in two tests, quality 92.5 % vs 91.9 %, and the earlier Grace 0.3.7 on 30 of those tasks, 95 % vs 94 % (with Grace vs Claude alone) — are at the top of the same page.
What this does not cover
The claims above are only worth something if their limits stand next to them.
- Small task sets. Each number comes from a small set of tasks — the count stands next to every number. In the subscription test, 3 of 34 tasks were left out of the comparison; the measurements page names them and says why.
- Judged by reading, not by running. In the subscription test, the reviewer read each result’s code against the criteria and did not run it. A criterion judged met can still fail when the code runs.
- Passing tests are not correctness. If a project’s tests do not cover the behaviour you care about, a green result in Grace’s report does not either. And where no tests can run, there is no check at all.
- Acceptance criteria are not your intent. A result can meet every criterion and still not be the change you wanted. That is a review, and a review is yours.
- New builds with Grace can trade a little. In the tests of October 2 – 3, 2026, new builds with Grace 0.3.8 (subscription runs) reached 92.5 % vs 91.9 % (Grace vs Claude alone) on the 58 of 60 tasks both runs finished; on 2 of them Grace was clearly worse (more than 10 points): G020, E004. In the subscription fill test their quality was slightly below Claude alone (91 % vs 94 %), furthest behind on the hardest tasks.
Questions
Does Grace lower quality to save output?
In the tests of October 2 – 3, 2026, new builds with Grace 0.3.8 (subscription runs) reached 92.5 % vs 91.9 % (Grace vs Claude alone) on the 58 of 60 tasks both runs finished; on 2 of them Grace was clearly worse (more than 10 points): G020, E004. On changes to existing code there is only an earlier measurement, in API mode on an interim build of September 28, 2026 (not the release version): Grace’s results met 95–100 % of the acceptance criteria on 6 tasks. In the subscription fill test, quality was slightly below Claude alone — 91 % vs 94 % on the 31 tasks both runs finished — and furthest behind on the hardest tasks. The exact numbers, with task counts, are above and on the measurements page.
Who judges quality?
Each task is judged against its own acceptance criteria, written with the task and kept secret from both Claude and Grace. In the subscription test, a separate Claude Opus 5.5 session judged every result by reading its code against the criteria: met counts 1, partly met ½, not met 0.
Does Grace check its own work?
When Grace builds a new project, its short report at the end says whether the project’s tests passed — or that there were no tests to run. That is no substitute for your own review. The quality numbers on this site are judged separately, against secret criteria.