The Real Cost of Frontier Models: Why $/Mtok is a Misleading Metric
The Real Cost of Frontier Models: Why $/Mtok is a Misleading Metric
The $/Mtok Metric is Not Comparable Across Vendors
The listed price per million tokens ($/Mtok) is not a constant because every model vendor uses a different tokenizer to convert raw text into billable units. Because a "token" is not a fixed amount of text, two models with the same list price can produce vastly different bills for the same input. The actual cost of a request is the product of the number of tokens the content becomes and the price per token.
Claude's New Tokenizer Increases Effective Costs
Anthropic's newest tokenizer (used in Sonnet 5, Opus 4.8, and Fable 5) produces approximately 32% more tokens for typical English and code requests than its previous version, despite the list prices remaining unchanged for some models. This effectively acts as a price increase without a corresponding line item on invoices.
Tokenization Inflation by Content Type
Measurements using Anthropic's own count_tokens endpoint show the following increases when moving from the old tokenizer to the new one:
| Content | Change in Token Count |
|---|---|
| English prose | +34% |
| TypeScript | +31% |
| Rust | +29% |
| Python | +23% |
| JavaScript | +20% |
| JSON tool schema | +26% |
| Agent system prompt | +39% |
| Chinese prose | ~0% |
The Sonnet 5 Pricing Window
Sonnet 5 launched with an introductory price of $2.00 / $10.00 (down from Sonnet 4.6's $3.00 / $15.00). While this lower rate covers the increased token count, the price is scheduled to return to $3.00 / $15.00 on September 1, 2026. After this date, the same workload will cost roughly one-third more than it did on Sonnet 4.6.
Cross-Vendor Tokenization Divergence
When using GPT-5.x's o200k_base tokenizer as a 1.00x reference, Claude's new tokenizer is significantly less efficient, particularly for code. This gap is widest in TypeScript, where Claude produces 1.73x more tokens than GPT for the same file.
Tokenization Multipliers (Claude New vs. GPT-5.x)
| Content | Multiplier |
|---|---|
| TypeScript | 1.73x |
| Rust | 1.58x |
| JavaScript | 1.52x |
| Python | 1.50x |
| English prose | 1.40x |
| Chinese prose | 1.44x |
TypeScript is the worst case because GPT's o200k is highly optimized for web-based languages, compressing camelCase identifiers and JSX patterns more efficiently than Claude's tokenizer.
Effective Pricing for Coding Workloads
The "effective price" is the list price multiplied by the tokenizer divergence. For a typical English coding request, the effective costs diverge significantly:
| Model | List Price (In/Out) | Divergence | Effective Price (In/Out) |
|---|---|---|---|
| GPT-5.1 | $1.25 / $10.00 | 1.00x | $1.25 / $10.00 |
| Gemini 3 Flash | $0.50 / $3.00 | 1.09x | $0.55 / $3.27 |
| Claude Sonnet 5 (Post-Sept 1) | $3.00 / $15.00 | 1.50x | $4.50 / $22.50 |
| Claude Opus 4.8 | $5.00 / $25.00 | 1.50x | $7.50 / $37.50 |
| Claude Fable 5 | $10.00 / $50.00 | 1.50x | $15.00 / $75.00 |
Beyond Input Tokenization: The Total Task Cost
Input tokenization is only one component of the total bill; model verbosity, reasoning effort, and caching strategies can cause total task costs to diverge far more than the 1.73x input gap.
Key Cost Drivers
- Cache Traffic: Since cache reads and writes are billed per token, a less efficient tokenizer increases the cost of every cache operation by the same percentage as the input inflation.
- Verbosity and Thinking: The number of output and "thinking" tokens a model generates to reach a solution can vary wildly. Some models may use 2-4x the tokens of others for the same agentic task due to internal reasoning loops or higher verbosity.
- Agentic Behavior: The frequency of tool calls, the amount of context loaded per step, and the use of sub-agents significantly impact the final invoice.
Community Insights
Users and developers have noted that these differences are felt most acutely in large codebases. One user reported a legacy C++ codebase of ~90k lines of code resulting in 1.12M tokens on GPT versus 2.2M on Claude. Others emphasize that the only way to accurately predict costs is to measure "dollars per completed task" rather than relying on rate cards.
Recommendations for Comparing Model Costs
- Test on Representative Content: Run a sample of your specific language and file types through each vendor's tokenizer to find your personal multiplier.
- Monitor Tokenizer Changes: Treat a tokenizer update as a price change. A model update (e.g., Opus 4.6 to 4.8) can result in a ~32% cost increase even if the list price is unchanged.
- Measure Task-Based Costs: Use the provider's
usagefield to calculate the cost per completed task, which captures tokenization, verbosity, and caching in one metric.