Migrating a Production AI Agent to GPT-5.6 Sol

Ploy has migrated its production AI agent—which plans, builds, and edits marketing websites—from Claude Opus 4.8 to GPT-5.6 Sol. The transition resulted in a 2.2x reduction in wall-clock time for builds and a 27% decrease in operational costs, while maintaining or improving visual quality scores.

Optimizing the Evaluation Harness for Model Parity

Evaluation harnesses are often silently tuned to the incumbent model, leading to inaccurate performance data when testing a challenger model. Ploy discovered that roughly one-third of initial failures for GPT-5.6 were caused by harness assumptions rather than model deficiencies.

Key Harness Failures

• Tool-Call Budgets: The existing harness was sized for the sequential tool-calling style of Claude Opus. GPT-5.6 utilizes parallel tool calls, which exhausted these budgets even when the model was solving the case correctly.
• Executor Support: The evaluation executor did not support batched file reads, a feature GPT-5.6 uses frequently but Opus rarely did.
• Implicit Thresholds: Some datasets lacked an explicit minScore threshold, defaulting to 1.0. This caused GPT-5.6 to "fail" high-quality outputs (e.g., scoring 0.98) that would have passed under a more flexible threshold.

Performance Comparison: Claude Opus 4.8 vs. GPT-5.6

Mean per completed build	Claude Opus 4.8 (n=11)	GPT-5.6 (n=10)
Cost	$3.06	$2.22
Wall-clock time	8m 00s	3m 42s
Input tokens	2.60M	1.70M
Output tokens	33.0K	17.1K
Visual score	0.936	0.970

Solving Tool Call Corruption via Schema Transformation

GPT-5.6 exhibits a specific behavior where it populates all optional parameters in a tool call with invented values (e.g., offset: 0) rather than omitting them. This makes invented values indistinguishable from intended arguments, leading to significant failures in tool execution.

The "Invented Value" Problem

In Ploy's code tool, which has 25 parameters, GPT-5.6 sent all 25 keys in 100% of calls, whereas Claude Opus 4.8 omitted unused parameters in 99.9% of cases. This resulted in 52% to 64% of file reads returning empty because the model inadvertently sent offset: 0 as a real argument.

The Fix: Required-but-Nullable Schemas

Prompting and OpenAI's strict mode did not resolve this behavior. Ploy implemented a schema transform at the provider boundary:

1. Rewrite Optional to Nullable: Every optional property is rewritten as required but nullable using anyOf: [T, null]. This forces the model to explicitly state null when a parameter is not used.
2. Strip Nulls: At the tool invocation seam, null values are stripped before validation, ensuring the tool implementation receives the same input it always did.

This change reduced empty file reads to 0% and decreased the total number of tool calls required per task by approximately 30%.

Reconfiguring Prompt Caching for GPT-5.6

Prompt caching on GPT-5.6 differs fundamentally from Anthropic's organization-scoped caching. Failure to correctly configure the prompt_cache_key can lead to a 0% cache hit rate and significantly higher costs.

Architectural Differences

• Anthropic (Claude): Caching is organization-scoped. A static prefix (e.g., 29K tokens of tool schemas) is cached once and shared across all conversations and workspaces.
• OpenAI (GPT-5.6): Caching requires explicit prompt_cache_breakpoint markers and a prompt_cache_key. Each key maps to a cache node that supports approximately 15 requests per minute (RPM). If traffic exceeds this, requests spill to cold nodes, causing cache misses.

Implementation Strategy: Workspace-Scoped Keys

To balance hit rates and throughput, Ploy implemented workspace-scoped keys (ws:{workspaceId}):

• Entry A (Static Prefix): Shared across all sessions in a workspace, making the first call of a new session cheap.
• Entry B (Workspace Context): Shared across conversations in the same workspace; updates when workspace memory changes.
• Entry C (Session Chain): Implicit whole-prompt chain for append-only conversation turns.

After this reconfiguration, first-call cache hits increased from 0% to 83.7%, and total uncached input tokens dropped by 28%.

Addressing Reasoning Replay Failures

GPT-5.6's Responses API replays prior-turn reasoning as server-side item references by default. This caused intermittent Item 'rs_...' not found errors mid-conversation. Ploy resolved this by setting store: false, which forces the SDK to request encrypted reasoning content and replay it as self-contained blobs rather than pointers to server state.

Community Insights and Counterpoints

Industry practitioners have noted that models are rarely interchangeable in production. As one developer noted:

Think of the whole harness, prompt, and model as one system, not really with modular parts that can be swapped out if you care about optimal performance.

Other critiques focused on the specific design quality of GPT-5.6, with some users preferring the brand adherence of Claude Opus, while others argued that GPT-5.6 is essentially a repackaged version of previous models with minimal quality gains over older versions like Opus 4.6.