GLM 5.2 VAT Benchmark: AI Accuracy in UK SME Bookkeeping

GLM 5.2 achieves near-human accuracy in VAT return preparation

GLM 5.2, an open-weights AI model, can prepare a nearly perfect quarterly value-added tax (VAT) return for a small UK business at a fraction of the cost of a human bookkeeper. In a benchmark involving 59 transactions, the model processed the data in 68 minutes with a raw token cost of $2.73, resulting in a net VAT position off by only 7 pence (~10 US cents) compared to human-prepared ground truth.

Benchmark Methodology

Data and Environment

Researchers used the first quarter of 2026 books from Vineyard Finance as the test set. The benchmark provided the model with three primary inputs for each transaction:

• Bank feed lines: JSON data containing date, amount, currency, and description.
• Receipt PDFs: Text-containing PDFs (no vision capabilities were required).
• User notes: Optional context (used in 2 of 59 cases) to provide real-world information not derivable from the bank feed or receipts.

GLM 5.2 was deployed on a Google Cloud Platform (GCP) instance via the Fireworks AI serverless tier. To ensure the model did not access the ground truth, it was isolated from the testing environment but granted internet access and access to a cloud-based accounting software via a pre-authenticated command-line interface (CLI) tool.

Scoring Criteria

Each transaction was evaluated based on six deterministic criteria derived from the final state of the accounting software:

1. Transaction Type: (e.g., purchase, sales income, transfer).
2. Category: The specific account from the chart of accounts.
3. VAT Treatment: (e.g., 20% VAT, 0% VAT, reverse charge).
4. VAT Amount: Within a 0.02 GBP tolerance.
5. Reverse-charge VAT: Within a 0.02 GBP tolerance.
6. Receipt Attachment: Verification that the correct evidence was attached.

Performance Analysis

Successes

GLM 5.2 demonstrated high reliability in several complex bookkeeping tasks:

• Accurate Classification: The model correctly assigned almost every transaction to the correct account in the chart of accounts.
• Disambiguation: It successfully handled tricky inputs, such as identical amounts from the same vendor on the same day, and transfers between company banks, and transactions split across multiple bank feed lines.
• Document Matching: The model never attached an incorrect invoice to a transaction.

Errors and Limitations

Out of 354 scored checks, the model failed 20 across 18 transactions. These errors fell into three categories:

1. Legal Classification Errors (Serious) The model incorrectly categorized "founding shares" (10,000 GBP) as a "Capital Account" instead of "Unpaid Shares." While this did not affect the VAT return, it has significant legal implications for company audits and end-of-year filings, as share capital is permanent, creditor-protecting capital.

2. Tax Category Confusion In 14 transactions, the model confused "zero-rated" VAT with "tax-exempt" VAT. Although neither involves a VAT payment, they are distinct categories in tax law. The model's performance here was stochastic, failing in January and February but succeeding in March.

**3. Reasoning Errors in Split Transactions In three instances involving multi-currency balances (Wise), the model occasionally "double-dipped" by accounting for VAT on both the main and residual legs of a split transaction, though the financial impact was immaterial.

Technical Resource Consumption

Month	Transactions	Turns	Tool Calls	Wall Time	Prompt Tokens	Output Tokens	Est. Cost
January	8	28	38	10.3 min	871,917	34,371	$0.45
February	29	37	44	31.4 min	1,873,745	65,929	$0.94
March	22	47	55	26.3 min	2,985,966	93,183	$1.34
Total	59	112	137	68 min	5.73M	193,483	$2.73

Note: Prompt tokens were 92-95% cached, significantly reducing costs.

Community Insights and Counterpoints

Discussion among technical users between benchmark success and 0.02 GBP tolerance in real-world deployment:

The "Last Mile" of Data Retrieval Critics noted that the benchmark provided the model with receipts and user notes, whereas a human bookkeeper a human bookkeeper must actively find invoices most often in mailboxes or request them from providers.

"Pretty much any non-entry office job worth having a having a lot of undocumented (even undocumentable) problems requiring judgment and and experience."

Liability and Accountability A recurring concern is the legal responsibility for tax errors. Unlike human accountants, LLMs cannot be held legally accountable or go to prison for tax fraud or negligence.

"You'2re not as much buying the service as you're buying not having to worry about the service."

The "Nearly Correct" Paradox Some users argued that that in tax compliance, "essentially correct" is insufficient, "essentially correct" is insufficient, as any error can lead to fines or audits.

"I can't file 'almost' with the IRS. That's not going to end well.