Migrating a Production AI Agent to GPT-5.6 Sol

Ploy has migrated its production AI agent from Claude Opus 4.8 to GPT-5.6 Sol, resulting in builds finishing in less than half the wall-clock time and a 27% reduction in cost. This transition demonstrates that moving between frontier models requires more than swapping an API key; it requires auditing the evaluation harness and resolving provider-specific behavioral differences in tool calling and prompt caching.

Optimizing the Evaluation Harness

Evaluation harnesses are often silently tuned to the incumbent model, leading to inaccurate performance metrics when testing a challenger model. Ploy discovered that roughly one-third of their initial raw failures with GPT-5.6 were caused by harness assumptions rather than model failures.

Addressing Model-Specific Behaviors

Tool-Call Budgets: GPT-5.6 utilizes parallel tool calls, whereas Claude Opus follows a more sequential style. Budgets sized for sequential calls caused GPT-5.6 to fail cases it was actually solving correctly.
Execution Support: GPT-5.6 frequently uses batched file reads, a feature the Ploy eval executor did not initially support, leading to false negatives.
Implicit Thresholds: Ploy found that missing minScore thresholds in their dataset defaulted to 1.0, causing models to fail cases where they scored 0.98 despite passing all individual checks.

Performance Comparison

After cleaning the harness, the head-to-head results for redesigning a brand's homepage showed significant gains for GPT-5.6:

Metric	Claude Opus 4.8 (n=11)	GPT-5.6 (n=10)
Cost	$3.06	$2.22
Wall-clock time	8m 00s	3m 42s
Input tokens	2.60M	1.70M
Output tokens	33.0K	17.1K
Visual score	0.936	0.970

Solving Tool-Call Parameter Inflation

GPT-5.6 exhibits a behavior where it sends all available parameters for a tool, even optional ones, inventing plausible values (e.g., offset: 0) instead of omitting them. This creates a critical failure mode where invented values are indistinguishable from intended ones.

The Impact of Invented Values

In Ploy's code tool, which has 25 parameters, GPT-5.6 sent all 25 properties in 100% of calls, compared to 0.1% for Claude Opus 4.8. This led to 52% to 64% of file reads returning empty because the model's invented offset: 0 was treated as a real instruction. Prompting and OpenAI's strict mode did not resolve this behavior.

The Schema Transform Fix

To resolve this, Ploy implemented a schema transform at the provider boundary for OpenAI models:

Rewrite Optional to Nullable: Every optional property is rewritten as required but nullable using anyOf: [T, null].
Explicit Nulls: This forces the model to explicitly state null for unused parameters.
Strip Nulls: The system strips these nulls before the tool is invoked, ensuring the tool implementation remains unchanged.

This fix reduced empty file reads to 0% and decreased the total number of tool calls by approximately 30%.

Reconfiguring Prompt Caching for GPT-5.6

Prompt caching implementations differ fundamentally between providers. Ploy found that GPT-5.6 was initially 50% more expensive than Opus due to cache misconfiguration.

Architectural Differences

Anthropic (Claude): Caching is organization-scoped. A static prefix is cached across all conversations and workspaces with high hit rates (92% to 96%).
OpenAI (GPT-5.6): Caching is based on prompt_cache_key. Identical prompts with different keys result in zero cache hits. Each key maps to a cache node supporting roughly 15 requests per minute (RPM).

Implementing Workspace-Scoped Caching

To avoid the 0% hit rate of per-conversation keys and the RPM bottlenecks of a single global key, Ploy adopted a per-workspace key strategy:

Shared Prefix: All conversations within a customer workspace share the same cache key.
Layered Breakpoints: The system prompt is split into breakpointed layers (Static Prefix $\rightarrow$ Workspace Context $\rightarrow$ Session Turns).

This configuration increased first-call cache hits from 0% to 83.7% and dropped uncached input tokens by 28%, bringing GPT-5.6's costs below those of Claude Opus.

Reasoning Replay and State Management

GPT-5.6's Responses API replays prior-turn reasoning as server-side item references by default. This caused intermittent Item 'rs_...' not found errors mid-conversation. Ploy resolved this by setting store: false, which forces the SDK to request encrypted reasoning content and replay it as self-contained blobs rather than pointers to server state.

Community Perspectives on Model Migration

Discussion among developers suggests that models are not truly interchangeable in production environments.

"Any production harness doing serious agentic work in production is dependent on more model-specific quirks than you would expect... Think of the whole harness, prompt, and model as one system, not really with modular parts that can be swapped out."

While Ploy reported significant gains, some developers expressed skepticism regarding the quality of GPT-5.6 compared to older versions or other models, noting that GPT-5.6 can converge toward a generic design aesthetic unless heavily steered.