For four months, no frontier model beat Claude Opus in our production evals. GPT-5.6 did. This is what we learned while migrating.
As of today, Ploy’s agent runs on GPT-5.6 Sol, the flagship tier of the model family OpenAI released this morning. For months, no model met our bar for replacing Claude Opus. GPT-5.6 Sol did. After a head-to-head evaluation, we made it the default model for every Ploy workspace.
Ploy’s agent builds and edits production marketing websites. It plans a page, reads the codebase, writes components, generates imagery, screenshots its own work, and decides when it is done. We test every frontier release against that workload. During the four months Opus held the default slot (first Opus 4.7, then 4.8), nothing we tested beat it. GPT-5.6 is the first model that did.
The first eval run exposed several failure modes, which we cover below. It also produced strong results. Completed builds took less than half the wall-clock time, cost 27% less, and scored at or above our incumbent. Those results justified the migration work.
We use Vercel’s AI SDK：https://ai-sdk.dev/docs/introduction, but switching from Claude Opus 4.8 to GPT-5.6 Sol still exposed provider-specific assumptions throughout our stack. The providers differ in how models fill tool arguments, cache prompts, and replay reasoning between turns.
We fixed the eval harness first, followed by the tool schemas, prompt caching, and reasoning replay.
Our eval suite runs the production agent against fixture workspaces. It covers hundreds of cases, from “build a homepage from scratch” to “is this clone request safe to execute.” For build cases, a visual judge runs ten binary checks against a reference design, such as “the hero is a full-bleed photographic scene” and “primary CTAs are rounded rectangles, not pills.” We also run content checks, tool-trajectory checks, and file assertions. We triage every failure against the full trace, including the tool calls and model text.
The first cross-model run exposed a problem in the eval harness.
Our tool-call budgets were sized for Opus’s sequential style. GPT-5.6 makes parallel calls and exceeded those budgets on cases it was solving correctly. Our eval executor also lacked support for batched file reads, which Opus rarely used and GPT-5.6 uses often. Roughly a third of the raw failures in the first run came from harness assumptions rather than model behavior, and those failures were unevenly distributed. If you are evaluating a challenger against an incumbent, triage the traces before trusting the pass rate. Otherwise, the eval rewards the new model for behaving like the old one.
A dataset that omitted its minScore threshold inherited a default of 1.0, and nothing flagged the fallback. GPT-5.6 therefore “failed” a hero that scored 0.98, while Opus “failed” a case despite passing every individual check. Both designs were defensible, but the implicit threshold failed them.
After fixing the harness, we reran our redesign suite, where the agent rebuilds a brand’s homepage against a reference design.
GPT-5.6 finished pages 2.2× faster, cost 27% less, and used about half the output tokens. It also wrote less code. In one matched pair, Opus produced a 17,957-character globals.css with 174 CSS variables, including mostly unused color ramps. GPT-5.6 used 2,508 characters and 45 variables for a comparable, and sometimes better, rendered page.
GPT-5.6 is good at clean, tightly gridded layouts, but it tends to converge on that style without strong steering. With our older harness, which was designed for Opus 4.8, GPT-5.6 Sol often ignored the existing design system and produced clean but generic output.
The changes we made here deserve a separate post. Our design and engineering teams improved the steering until GPT-5.6 met the brand-adherence bar we require in production.
Our agent’s code tool has 25 top-level parameters. One, action , is required; the rest are optional. Claude sends the two or three parameters it uses and omits the rest. GPT-5.6 sends all 25 on every call, filling unused parameters with plausible values such as offset: 0 , timeout: 120000 , and siteId: "00000000-0000-0000-0000-000000000000" .
We saw the pattern across three days of production code(read) traces.
Verbosity was not the problem. The file-read implementation could not distinguish an invented value from an intended one. It treated offset: 0 as a real argument, causing 52% to 64% of GPT-5.6’s file reads to return empty. The tool returned success: true for both valid and empty reads, so the model could not tell that it was reading blank files. It compensated with more calls and worse results.
Prompting did not fix this. A tool-description directive to “omit unused parameters” still produced 25 of 25 parameters. Per-property “OPTIONAL, omit if unused” hints did the same. We measured identical behavior with OpenAI’s strict mode, and adopting it would have required stripping pattern , format , and array-bound validation from every schema. This behavior comes from how the model emits function calls：https://developers.openai.com/api/docs/guides/function-calling, so we changed the schema instead.
The working fix was a schema transform at the provider boundary. For OpenAI-family models, we rewrite every optional property as required but nullable using anyOf: [T, null] . This gives the model an explicit value for an unused parameter. We then strip the nulls before validation at the shared tool-invocation boundary, so the tool implementations do not change. The model sees a schema that represents unused values, while the tools receive the same inputs as before.
After the change, empty file reads fell from 52% to 0%. The agent also used roughly 30% fewer tool calls for the same work because it stopped re-reading blank results.
Both providers offer “prompt caching,” but the implementations differ. Before we accounted for those differences, GPT-5.6 appeared about 50% more expensive than Opus. Our cache configuration caused the gap. The model pricing was fine.
Our agent’s prompt opens with a static prefix of roughly 29K tokens (tool schemas plus the core system prompt) that is identical for every conversation. On Claude, we mark cache breakpoints with cache_control , and the prefix caches across the whole organization. Any conversation in any workspace can use one shared entry, without a per-key throughput budget. Cache hit rates run from 92% to 96%.
GPT-5.6 uses a different OpenAI caching model. Earlier GPT models cached implicit partial-prefix matches. GPT-5.6 dropped partial-prefix matching：https://developers.openai.com/api/docs/guides/prompt-caching, so implicit caching now creates whole-prompt entries keyed on the latest message. A new conversation sharing our 29K static prefix cached 0% of it. Each conversation re-billed the full prefix at the uncached rate. GPT-5.6 also applies a 1.25× cache-write surcharge to every uncached prompt, whether or not the application uses caching.
The explicit mechanism uses prompt_cache_breakpoint markers and a mandatory prompt_cache_key . The key is part of the cache identity, so identical prompts with different keys produce no cache hits. Each key maps to a cache node that sustains roughly 15 requests per minute before OpenAI distributes traffic to other nodes with independent cold caches.
The main design decision is which entity should scope the key.
We ship the workspace-scoped key and split the system prompt into breakpointed layers, mirroring the structure we already used for Anthropic:
Entry A reduces the cost of a session’s first call. When workspace memory changes, the request misses Entry B but still hits Entry A, then writes a new Entry B. That requires one context-sized write instead of re-billing the full 29K prefix. Entry C is OpenAI’s implicit whole-prompt chain, which works within a session because our prompts are strictly append-only.
OpenAI’s key partitioning prevents cross-workspace sharing of the static prefix. Anthropic can share the prefix because its cache is organization-scoped without key partitioning. On GPT-5.6, every workspace pays for one 29K cold write per idle window, about $0.18. The cost is bounded and predictable.
After the change, first-call cache hits rose from roughly 0% to 83.7%, total uncached input tokens fell 28%, and GPT-5.6’s per-suite cost dropped below Opus’s. Cache misconfiguration accounted for the entire cost gap we had measured. Cost comparisons between models are not useful when one model starts with a cold cache.
Reasoning replay caused intermittent failures in production conversations. GPT-5.6’s Responses API replays prior-turn reasoning as server-side item references by default, and our conversations sometimes failed with Item 'rs_...' not found . Setting store: false makes the SDK request encrypted reasoning content：https://developers.openai.com/api/docs/guides/reasoning and replay self-contained blobs instead of pointers to server state. We also learned that server-side reasoning state can change the effective prompt even when the bytes sent by the application are append-only.
GPT-5.6 launched today and is already live on Ploy. You can start free at ploy.ai：https://ploy.ai/auth/sign-up, give it a website to build, and see what a sub-four-minute build looks like.
Ploy is an AI layer that plans, builds, publishes, and optimizes websites and campaigns. If debugging cache-node fan-out at 2am sounds like fun, we’re hiring：https://ploy.ai/careers.
Ploy builds, tests, and optimizes your website. Automatically.
Scan to open this article