GLM 5.2 Achieves Near‑Human Accuracy on UK VAT Return Benchmark

TL;DR

GLM 5.2 generated a quarterly VAT return for a UK SME with a net VAT figure off by just 7 pence, at a raw token cost of $2.73 and a wall‑clock time of 68 minutes for 59 transactions, proving that open‑weight LLMs can perform bookkeeping tasks at a fraction of traditional accountant fees.

Benchmark Scope and Economic Context

Preparing a UK VAT return is a mandatory quarterly compliance task for VAT‑registered SMEs.
Professional accountants typically charge £750–£2,100 per quarter (≈ $1,000–$2,800).
The benchmark measured whether GLM 5.2 could replace that service by processing raw transaction data and receipts, then populating a cloud‑based accounting system via a CLI.

Test Setup and Isolation

Data source – First‑quarter 2026 books of Vineyard Finance, extracted with Claude Fable 5. The data included bank‑feed JSON lines, text‑only PDF receipts, and two optional user notes ("founder shares" and "personal car hire").
Execution environment – GLM 5.2 ran on a dedicated GCP VM with internet access but no direct access to the ground‑truth books. The model could only use two tools: a Bash executor and a session‑termination/reporting hook.
Model provider – Fireworks AI serverless tier (quantisation undisclosed, likely FP16 or FP8). No evidence of cheating was found; the only external lookup was a query about reverse‑charge VAT specific to the accounting software.

Scoring Methodology

Each transaction was evaluated on six criteria after the model finished its run:

Transaction type (purchase, bank fee, transfer, etc.)
Chart‑of‑accounts category
VAT treatment (reverse charge, standard 20 %, zero‑rated, exempt)
VAT amount (± £0.02 tolerance)
Reverse‑charge VAT amount (± £0.02 tolerance)
Receipt attached (mandatory for tax compliance)

A total of 354 checks (59 × 6) were performed.

Performance Metrics

Period	Transactions	Turns (API calls)	Tool calls	Wall time	Prompt tokens	Output tokens	Peak context usage	Estimated cost
January	8	28	38	10.3 min	871,917	34,371	66,381 (~6 % of 1 M)	$0.45
February	29	37	44	31.4 min	1,873,745	65,929	111,246 (~11 %)	$0.94
March	22	47	55	26.3 min	2,985,966	93,183	139,128 (~13 %)	$1.34
Quarter	59	112	137	68 min	5.73 M	193,483	139,128	$2.73

*Each "turn" represents a full conversation resend, which explains the multi‑million prompt token count. Over 90 % of prompts were served from the provider’s cache, reducing cost.

Accuracy Outcomes

Overall error rate: 20 failures across 354 checks (≈ 5.6 %).
Serious error: Mis‑classification of founder share capital as a "Capital Account" instead of the legally required "Unpaid Shares". This could affect statutory filings and audit risk, though it did not change the VAT figure.
Common minor errors:
- Confusing zero‑rated with tax‑exempt VAT categories in 14 transactions (no financial impact but a bookkeeping best‑practice violation).
- Double‑counting VAT on split‑leg Wise transactions in 3 cases; one instance was corrected by the model, the other two were counted as errors by the conservative scorer.
Critical success metric: The net VAT payable (Box 5) differed by only £0.07 (≈ $0.10) from the human‑prepared return.

Areas of Perfect Performance

Correctly assigned every transaction to the appropriate chart‑of‑accounts category (except the share‑capital mistake).
Always attached the correct receipt PDF to each transaction.
Disambiguated tricky scenarios such as:
- Identical‑amount, same‑vendor, same‑day entries.
- Transfers between the company’s own bank accounts.
- Card purchases that were actually internal transfers.
Handled multi‑currency split transactions without human intervention.

Implications for Book‑Keeping Automation

Cost efficiency: At <$3 per quarter, GLM 5.2 is < 1 % of typical accountant fees.
Speed: 68 minutes to process a full quarter of activity is comparable to, or faster than, manual data entry.
Regulatory risk: While the VAT figure is essentially spot‑on, the share‑capital mis‑classification shows that legal‑entity accounting rules still require human oversight.
Scalability: The benchmark used a minimal toolset (bash + reporting). Adding domain‑specific knowledge bases (e.g., UK tax legislation) could eliminate the zero‑rated vs. exempt confusion.

Community Reactions (Selected HN Comments)

"The job performed by the humans was broader than what was requested of the model… any non‑entry office job involves undocumented problems requiring judgment and experience." – Diogenesian

"If an LLM decides to do a little bit of tax fraud, you’re in uncharted waters. The gun did it, but you were the one holding the gun." – zerobees

"Humans aren’t exactly known for perfect recall. … What is this metric?" – malfist

"I wouldn’t be surprised if it were more accurate based on the errors I’ve seen… layers and multiple sets of eyes are still important." – petercooper

"This doesn’t surprise me at all. You can really constrain this problem, give very narrow context, and get pretty reliable results." – traverseda

These comments highlight two recurring themes: the need for human oversight on edge cases and the potential for LLMs to surpass human accuracy on routine, well‑structured tasks.

Next Steps for Practitioners

Integrate domain‑specific tax rule engines to resolve zero‑rated vs. exempt distinctions automatically.
Add validation layers that flag high‑impact classification errors (e.g., share‑capital handling) for human review before final filing.
Leverage caching and token‑reuse as demonstrated to keep operational costs low.
Pilot the workflow on live SME accounts to measure real‑world reliability and compliance with HMRC audit standards.

Book‑keeping is rapidly approaching a solved problem for routine compliance tasks. The remaining challenge is building safe, auditable scaffolding that lets SMEs reap the cost and speed benefits while preserving legal compliance.