Ploy 将 AI 智能体默认模型从 Claude Opus 4.8 切换至 GPT-5.6 Sol

在四个月的时间里，没有任何前沿模型在我们的生产评估中超过 Claude Opus。GPT-5.6 做到了。这是我们在迁移过程中学到的东西。

截至今天，Ploy 的代理运行在 GPT-5.6 Sol 上，这是 OpenAI 今天上午发布的模型家族的旗舰级别。在几个月的时间里，没有模型达到我们替换 Claude Opus 的标准。GPT-5.6 Sol 做到了。在一次正面评价之后，我们将其设为每个 Ploy 工作区的默认模型。

Ploy 的代理构建和编辑生产营销网站。它规划页面，读取代码库，编写组件，生成图像，为自己的工作截图，并决定何时完成。我们测试每一次前沿发布版本时都会使用这个工作负载。在 Opus 占据默认槽的四个月期间（先是 Opus 4.7，然后是 4.8），我们测试的所有模型都无法超过它。GPT-5.6 是第一个做到这一点的模型。

首次评估运行暴露了几个失败模式，我们将在下面介绍。它也产生了强劲的结果。完成的构建所需时间不到实际时钟时间的一半，成本降低了 27%，并且评分达到或超过了我们现有模型。这些结果证明了迁移工作的合理性。

我们使用 Vercel 的 AI SDK：https://ai-sdk.dev/docs/introduction，但从 Claude Opus 4.8 切换到 GPT-5.6 Sol 仍然暴露了整个技术栈中供应商特定的假设。供应商在模型填充工具参数、缓存提示以及在回合间重放推理的方式上存在差异。

我们首先修复了评估框架，然后是工具模式、提示缓存和推理重放。

我们的评估套件将生产代理运行在固定工作区上。它涵盖了数百个用例，从“从零构建首页”到“此克隆请求是否可以安全执行”。对于构建用例，视觉评审会针对参考设计进行十项二元检查，例如“主图是全幅摄影场景”和“主要 CTA 是圆角矩形，而不是胶囊状”。我们还进行内容检查、工具轨迹检查和文件断言。我们针对完整追踪包括工具调用和模型文本，对每个失败进行分类。

首次跨模型运行暴露了评估框架中的一个问题。

我们的工具调用预算是按照 Opus 的顺序调用风格设计的。GPT-5.6 执行并行调用，并在正确解决问题的情况下超出了这些预算。我们的评估执行器也不支持批量文件读取，而 Opus 很少使用这一功能，而 GPT-5.6 经常使用。第一次运行中约三分之一的原始失败来自测试假设，而不是模型行为，这些失败分布不均。如果你正在评估一个挑战者与现任模型的表现差异，在信任通过率之前要先排查日志。否则，评估会奖励新模型表现得像旧模型。

一个省略 minScore 阈值的数据集继承了默认值 1.0，但没有任何东西标记了这一回退值。因此，GPT-5.6 “未通过”一个得分为 0.98 的 hero，而 Opus 即便通过了每个单独检查也“未通过”某个案例。两者的设计都是合理的，但隐式阈值让它们失败了。

在修复测试工具后，我们重新运行了重新设计测试套件，其中代理根据参考设计重建一个品牌的主页。

GPT-5.6 完成页面的速度快了 2.2 倍，成本低了 27%，输出的 token 量约为一半。同时，它写的代码也更少。在一个匹配示例中，Opus 生成了一个 17,957 字符的 globals.css，包含 174 个 CSS 变量，其中大部分颜色梯度未被使用。GPT-5.6 用 2,508 个字符和 45 个变量生成了一个可比较、有时甚至更好的渲染页面。

GPT-5.6 擅长干净、紧凑网格布局，但如果没有强力引导，它容易收敛到这种风格。使用我们设计给 Opus 4.8 的旧测试工具时，GPT-5.6 Sol 经常忽略现有的设计系统，生成干净但通用化的输出。

我们在这里做的更改值得单独发一篇文章。我们的设计和工程团队改进了引导方法，直到 GPT-5.6 达到我们生产中要求的品牌一致性标准。

我们代理的代码工具有 25 个顶级参数。其中一个 action 是必需的，其余都是可选的。Claude 只发送它使用的两个或三个参数，省略其余参数。GPT-5.6 每次调用发送全部 25 个参数，将未使用的参数填充为合理值，例如 offset: 0、timeout: 120000 和 siteId: "00000000-0000-0000-0000-000000000000"。

我们在三天的生产代码（读取）日志中都看到了这个模式。

冗长并不是问题。文件读取的实现无法区分一个虚构的值和一个预期值。它将 offset: 0 当作真实参数处理，导致 GPT-5.6 有 52% 到 64% 的文件读取返回为空。该工具对于有效和空读取都返回 success: true，因此模型无法判断自己正在读取空文件。它通过更多的调用来补偿，但结果更差。

提示无法解决这个问题。一个提示工具描述指令“省略未使用的参数”仍然生成了 25 个参数中的 25 个。针对每个属性的“OPTIONAL，如果未使用则省略”提示也是一样。我们在 OpenAI 的严格模式中测量到相同的行为，采用严格模式将需要从每个模式中去掉 pattern、format 和 array-bound 验证。这种行为来源于模型发出函数调用的方式：https://developers.openai.com/api/docs/guides/function-calling，因此我们改为了修改模式。

有效的修复是在提供者边界进行模式转换。对于 OpenAI 系列模型，我们将每个可选属性重写为必需但可为空，使用 anyOf: [T, null]。这为模型提供了未使用参数的明确值。然后我们在共享工具调用边界的验证前去掉 null，因此工具实现无需改变。模型看到的模式表示未使用的值，而工具收到的输入与之前相同。

更改后，空文件读取从 52% 降至 0%。由于停止了重新读取空结果，代理在完成相同任务时调用工具的次数也减少了约 30%。

两个提供商都提供“提示缓存”，但实现方式不同。在考虑这些差异之前，GPT-5.6 的使用成本似乎比 Opus 高约 50%。造成差异的是我们的缓存配置，模型定价本身是正常的。

我们代理的提示以大约 29K 令牌的静态前缀开头（工具模式加核心系统提示），每次对话都相同。在 Claude 上，我们用 cache_control 标记缓存中断点，前缀可以在整个组织中共享。任何工作区的任何对话都可以使用一个共享条目，无需每个密钥的吞吐量预算。缓存命中率在 92% 到 96% 之间。

GPT-5.6 使用了不同的 OpenAI 缓存模型。早期的 GPT 模型会缓存隐式的部分前缀匹配。GPT-5.6 放弃了部分前缀匹配：https://developers.openai.com/api/docs/guides/prompt-caching，因此现在隐式缓存会根据最新消息创建完整提示条目。一个新的会话共享我们 29K 的静态前缀时，缓存了 0%。每个会话都会按照未缓存的费率重新计费完整前缀。GPT-5.6 还会对每个未缓存的提示应用 1.25× 的缓存写入附加费，无论应用是否使用缓存。

显式机制使用 prompt_cache_breakpoint 标记和强制性 prompt_cache_key。该键是缓存标识的一部分，因此相同提示但键不同不会产生缓存命中。每个键映射到一个缓存节点，该节点在 OpenAI 将流量分发到其他拥有独立冷缓存的节点之前，大约每分钟可支持 15 个请求。

主要的设计决策是哪个实体应该负责限定键的范围。

我们提供工作区范围的键，并将系统提示拆分为有断点的层，这与我们之前用于 Anthropic 的结构相对应：

Entry A 减少了会话首次调用的成本。当工作区内存发生变化时，请求会错过 Entry B，但仍命中 Entry A，然后写入一个新的 Entry B。这只需一次与上下文大小相等的写入，而不是重新计费完整的 29K 前缀。Entry C 是 OpenAI 的隐式完整提示链，它在会话内工作，因为我们的提示严格为追加模式。

OpenAI 的键分区防止了静态前缀在工作区之间的共享。Anthropic 可以共享前缀，因为其缓存是组织范围的，没有键分区。在 GPT-5.6 上，每个工作区在空闲窗口中需为一次 29K 的冷写入支付约 0.18 美元。成本是有限且可预测的。

更改之后，首次调用的缓存命中率从大约 0% 提升到 83.7%，未缓存输入令牌总数下降 28%，GPT-5.6 的每套成本低于 Opus。缓存配置错误导致了我们测量到的全部成本差异。当一个模型从冷缓存开始时，对模型之间的成本比较没有参考价值。

推理重放导致生产对话出现间歇性故障。GPT-5.6 的 Responses API 默认会将先前回合的推理作为服务器端项目引用进行重放，而我们的对话有时会出现 Item 'rs_...' not found 错误。将 store 设置为 false 可使 SDK 请求加密的推理内容：https://developers.openai.com/api/docs/guides/reasoning，并重放自包含的块而不是指向服务器状态的指针。我们还了解到，即使应用程序发送的字节是追加唯一的，服务器端的推理状态也可能改变有效提示。

GPT-5.6 今天上线，并已在 Ploy 上运行。你可以在 ploy.ai 免费开始：https://ploy.ai/auth/sign-up，提供一个网站以生成，并查看不到四分钟的生成效果。

Ploy 是一个 AI 层，可以规划、构建、发布并优化网站和活动。如果凌晨 2 点调试缓存节点分发听起来很有趣，我们正在招聘：https://ploy.ai/careers。

Ploy 自动构建、测试并优化你的网站。

For four months, no frontier model beat Claude Opus in our production evals. GPT-5.6 did. This is what we learned while migrating.

As of today, Ploy’s agent runs on GPT-5.6 Sol, the flagship tier of the model family OpenAI released this morning. For months, no model met our bar for replacing Claude Opus. GPT-5.6 Sol did. After a head-to-head evaluation, we made it the default model for every Ploy workspace.

Ploy’s agent builds and edits production marketing websites. It plans a page, reads the codebase, writes components, generates imagery, screenshots its own work, and decides when it is done. We test every frontier release against that workload. During the four months Opus held the default slot (first Opus 4.7, then 4.8), nothing we tested beat it. GPT-5.6 is the first model that did.

The first eval run exposed several failure modes, which we cover below. It also produced strong results. Completed builds took less than half the wall-clock time, cost 27% less, and scored at or above our incumbent. Those results justified the migration work.

We use Vercel’s AI SDK：https://ai-sdk.dev/docs/introduction, but switching from Claude Opus 4.8 to GPT-5.6 Sol still exposed provider-specific assumptions throughout our stack. The providers differ in how models fill tool arguments, cache prompts, and replay reasoning between turns.

We fixed the eval harness first, followed by the tool schemas, prompt caching, and reasoning replay.

Our eval suite runs the production agent against fixture workspaces. It covers hundreds of cases, from “build a homepage from scratch” to “is this clone request safe to execute.” For build cases, a visual judge runs ten binary checks against a reference design, such as “the hero is a full-bleed photographic scene” and “primary CTAs are rounded rectangles, not pills.” We also run content checks, tool-trajectory checks, and file assertions. We triage every failure against the full trace, including the tool calls and model text.

The first cross-model run exposed a problem in the eval harness.

Our tool-call budgets were sized for Opus’s sequential style. GPT-5.6 makes parallel calls and exceeded those budgets on cases it was solving correctly. Our eval executor also lacked support for batched file reads, which Opus rarely used and GPT-5.6 uses often. Roughly a third of the raw failures in the first run came from harness assumptions rather than model behavior, and those failures were unevenly distributed. If you are evaluating a challenger against an incumbent, triage the traces before trusting the pass rate. Otherwise, the eval rewards the new model for behaving like the old one.

A dataset that omitted its minScore threshold inherited a default of 1.0, and nothing flagged the fallback. GPT-5.6 therefore “failed” a hero that scored 0.98, while Opus “failed” a case despite passing every individual check. Both designs were defensible, but the implicit threshold failed them.

After fixing the harness, we reran our redesign suite, where the agent rebuilds a brand’s homepage against a reference design.

GPT-5.6 finished pages 2.2× faster, cost 27% less, and used about half the output tokens. It also wrote less code. In one matched pair, Opus produced a 17,957-character globals.css with 174 CSS variables, including mostly unused color ramps. GPT-5.6 used 2,508 characters and 45 variables for a comparable, and sometimes better, rendered page.

GPT-5.6 is good at clean, tightly gridded layouts, but it tends to converge on that style without strong steering. With our older harness, which was designed for Opus 4.8, GPT-5.6 Sol often ignored the existing design system and produced clean but generic output.

The changes we made here deserve a separate post. Our design and engineering teams improved the steering until GPT-5.6 met the brand-adherence bar we require in production.

Our agent’s code tool has 25 top-level parameters. One, action , is required; the rest are optional. Claude sends the two or three parameters it uses and omits the rest. GPT-5.6 sends all 25 on every call, filling unused parameters with plausible values such as offset: 0 , timeout: 120000 , and siteId: "00000000-0000-0000-0000-000000000000" .

We saw the pattern across three days of production code(read) traces.

Verbosity was not the problem. The file-read implementation could not distinguish an invented value from an intended one. It treated offset: 0 as a real argument, causing 52% to 64% of GPT-5.6’s file reads to return empty. The tool returned success: true for both valid and empty reads, so the model could not tell that it was reading blank files. It compensated with more calls and worse results.

Prompting did not fix this. A tool-description directive to “omit unused parameters” still produced 25 of 25 parameters. Per-property “OPTIONAL, omit if unused” hints did the same. We measured identical behavior with OpenAI’s strict mode, and adopting it would have required stripping pattern , format , and array-bound validation from every schema. This behavior comes from how the model emits function calls：https://developers.openai.com/api/docs/guides/function-calling, so we changed the schema instead.

The working fix was a schema transform at the provider boundary. For OpenAI-family models, we rewrite every optional property as required but nullable using anyOf: [T, null] . This gives the model an explicit value for an unused parameter. We then strip the nulls before validation at the shared tool-invocation boundary, so the tool implementations do not change. The model sees a schema that represents unused values, while the tools receive the same inputs as before.

After the change, empty file reads fell from 52% to 0%. The agent also used roughly 30% fewer tool calls for the same work because it stopped re-reading blank results.

Both providers offer “prompt caching,” but the implementations differ. Before we accounted for those differences, GPT-5.6 appeared about 50% more expensive than Opus. Our cache configuration caused the gap. The model pricing was fine.

Our agent’s prompt opens with a static prefix of roughly 29K tokens (tool schemas plus the core system prompt) that is identical for every conversation. On Claude, we mark cache breakpoints with cache_control , and the prefix caches across the whole organization. Any conversation in any workspace can use one shared entry, without a per-key throughput budget. Cache hit rates run from 92% to 96%.

GPT-5.6 uses a different OpenAI caching model. Earlier GPT models cached implicit partial-prefix matches. GPT-5.6 dropped partial-prefix matching：https://developers.openai.com/api/docs/guides/prompt-caching, so implicit caching now creates whole-prompt entries keyed on the latest message. A new conversation sharing our 29K static prefix cached 0% of it. Each conversation re-billed the full prefix at the uncached rate. GPT-5.6 also applies a 1.25× cache-write surcharge to every uncached prompt, whether or not the application uses caching.

The explicit mechanism uses prompt_cache_breakpoint markers and a mandatory prompt_cache_key . The key is part of the cache identity, so identical prompts with different keys produce no cache hits. Each key maps to a cache node that sustains roughly 15 requests per minute before OpenAI distributes traffic to other nodes with independent cold caches.

The main design decision is which entity should scope the key.

We ship the workspace-scoped key and split the system prompt into breakpointed layers, mirroring the structure we already used for Anthropic:

Entry A reduces the cost of a session’s first call. When workspace memory changes, the request misses Entry B but still hits Entry A, then writes a new Entry B. That requires one context-sized write instead of re-billing the full 29K prefix. Entry C is OpenAI’s implicit whole-prompt chain, which works within a session because our prompts are strictly append-only.

OpenAI’s key partitioning prevents cross-workspace sharing of the static prefix. Anthropic can share the prefix because its cache is organization-scoped without key partitioning. On GPT-5.6, every workspace pays for one 29K cold write per idle window, about $0.18. The cost is bounded and predictable.

After the change, first-call cache hits rose from roughly 0% to 83.7%, total uncached input tokens fell 28%, and GPT-5.6’s per-suite cost dropped below Opus’s. Cache misconfiguration accounted for the entire cost gap we had measured. Cost comparisons between models are not useful when one model starts with a cold cache.

Reasoning replay caused intermittent failures in production conversations. GPT-5.6’s Responses API replays prior-turn reasoning as server-side item references by default, and our conversations sometimes failed with Item 'rs_...' not found . Setting store: false makes the SDK request encrypted reasoning content：https://developers.openai.com/api/docs/guides/reasoning and replay self-contained blobs instead of pointers to server state. We also learned that server-side reasoning state can change the effective prompt even when the bytes sent by the application are append-only.

GPT-5.6 launched today and is already live on Ploy. You can start free at ploy.ai：https://ploy.ai/auth/sign-up, give it a website to build, and see what a sub-four-minute build looks like.

Ploy is an AI layer that plans, builds, publishes, and optimizes websites and campaigns. If debugging cache-node fan-out at 2am sounds like fun, we’re hiring：https://ploy.ai/careers.

Ploy builds, tests, and optimizes your website. Automatically.