AWS Billing Glitch Shows $1.7 Billion Estimated Charges – What Went Wrong

AWS Billing Glitch Shows $1.7 Billion Estimated Charges – What Went Wrong

TL;DR – Massive AWS billing estimate error

A worldwide AWS billing glitch showed some customers an estimated bill of $1.7 billion for a single month, far above the typical <$5 usage. AWS rolled back the estimates to the last known good values and opened a support ticket for affected users.

What happened: a unit‑conversion bug caused astronomic estimates

The root cause was a mis‑configured pricing unit. A line item that should have been priced per gigabyte (GB) was mistakenly set to price per byte. Because the metering system joins usage records to pricing plans, the wrong unit caused the system to multiply usage by a factor of 2³⁰ (≈1 billion). The result was estimated bills in the hundreds of billions of dollars for many accounts.

"In my case we meant to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults to bytes. 5¢ per Byte of data transferred meant some customers were seeing MM bills within hours." – donavanm (comment)

Immediate impact on customers

  • Users reported estimates ranging from $78 million to $595 billion, far exceeding any realistic spend.
  • The estimates triggered budget alerts, causing panic and, in some cases, threats to abandon AWS.
  • A handful of users posted screenshots of the erroneous numbers, confirming the scale of the issue.

"I logged in this morning to find a bill of $595 billion… then I noticed the open issue, phew!" – marksk

AWS response: rollback and communication

AWS acknowledged the problem on the AWS Health Dashboard and promised to revert estimates to the last known good state (as of July 15). The rollback restored reasonable numbers for most customers, though the exact timing varied across accounts.

"Will wait for the RCA, the update says that they will resort to last known estimate as of 15 July." – abkolan

Why the bug mattered beyond the headline numbers

  1. Financial forecasting – Companies rely on AWS cost forecasts for budgeting and investor reporting. An inflated estimate can jeopardize financing rounds or trigger unnecessary cost‑cutting measures.
  2. Trust in metering – The incident exposed how a single unit error can cascade through AWS’s massive metering pipeline, raising concerns about the robustness of pricing metadata.
  3. Operational overhead – Support teams had to field hundreds of urgent tickets, diverting resources from other critical incidents.

Lessons for cloud users

  • Enable cost anomaly detection: Set tight budget alerts and use AWS Budgets to catch spikes early.
  • Validate pricing units: When creating custom pricing plans or using third‑party services, double‑check the unit of measurement.
  • Monitor the AWS Health Dashboard: It provides real‑time status on service‑wide issues that can affect billing.

Community reaction

The Hacker News thread was filled with hyperbolic reactions (e.g., "$13 trillion bill, combined GDP of France, Germany, and UK") but also practical advice:

  • Some users shared their actual bills (e.g., $13,648,114,178,401.01) to illustrate the absurdity.
  • Others noted that similar bugs have occurred before, such as EC2 reservation‑savings miscalculations in the early 2010s.
  • A few suggested the issue might have been amplified by a flawed estimation algorithm that required prior estimates to compute the current one, leading to division‑by‑zero errors.

"Why does AWS need to roll back estimated bills to a 'last known good' state? ... I guess the problem is that someone decided to get too fancy with the estimates, and built a latent requirement that all prior estimates for the month must be available to compute the current estimate." – simonreiff

Outlook

AWS has a strong track record of fixing billing anomalies within hours, and the company’s swift rollback suggests the issue was isolated to the pricing metadata layer rather than a systemic failure. Nonetheless, the incident serves as a reminder that even the world’s largest cloud provider can suffer from simple configuration errors that have outsized financial implications.

All statements are based on the original Hacker News post and the comments linked therein. No additional sources were consulted.

Sources