GLM‑5.2 VAT Benchmark: Near‑Human Accuracy for a Fraction of the Cost

GLM‑5.2 VAT Benchmark: Near‑Human Accuracy for a Fraction of the Cost

GLM‑5.2 can prepare a UK VAT return with sub‑penny accuracy for a fraction of the cost of a human accountant

Takeaway: In a controlled benchmark GLM‑5.2 produced a quarterly VAT return that was off by only £0.07 (≈ $0.10) compared with a human‑prepared return, while costing just $2.73 in raw token usage – less than 1 % of the typical £750‑£2,100 quarterly fee charged by accounting firms.

Benchmark design and environment

The test measured how well GLM‑5.2 could automate the entire VAT‑return workflow for a small UK business.

  • Data source: The first quarter of Vineyard Finance’s 2026 books (January‑March 2026). Transactions and receipts were extracted with Claude Fable 5 and presented to the model as JSON bank‑feed lines, text‑only PDF receipts, and occasional “user notes” (e.g., founder shares, personal car hire).

  • Execution platform: GLM‑5.2 ran on a Google Cloud Platform VM isolated from the benchmark data store. The model accessed the cloud‑based accounting SaaS via a pre‑authenticated CLI tool and could call the internet only for operational API requests. No external knowledge bases were injected.

  • Tooling: The model was limited to a bash‑style tool for interacting with the accounting CLI and a termination/reporting tool. No vision capabilities were needed because all receipts were text‑based PDFs.

  • Scoring methodology: After the model processed each transaction, the final state of the accounting software was inspected against six criteria per transaction:

    1. Transaction type (purchase, sales, transfer, etc.)
    2. Chart‑of‑accounts category
    3. VAT treatment (standard, reverse‑charge, exempt, etc.)
    4. VAT amount (± £0.02 tolerance)
    5. Reverse‑charge VAT amount (± £0.02 tolerance)
    6. Presence of a linked receipt

    A total of 354 checks (59 transactions × 6 criteria) were evaluated.

Performance metrics

Period Transactions Turns (API calls) Tool calls Wall‑time Prompt tokens Output tokens Peak context usage Estimated cost
January 8 28 38 10.3 min 871 917 34 371 66 381 (6.3 % of 1 048 576) $0.45
February 29 37 44 31.4 min 1 873 745 65 929 111 246 (10.6 %) $0.94
March 22 47 55 26.3 min 2 985 966 93 183 139 128 (13.3 %) $1.34
Quarter 59 112 137 68 min 5.73 M 193 483 139 128 $2.73

*Each month ran as a single continuous agent session; a “turn” corresponds to one API request, and the full conversation history was resent each turn, leading to multi‑million‑token prompts. Over 90 % of prompt tokens were served from the provider’s cache, reducing the effective price.

Accuracy results

  • Overall error rate: 20 failed checks across 18 transactions (5.6 % of the 354 checks).
  • Financial impact: The net VAT position (Box 5) differed by only £0.07 from the human‑verified return – effectively negligible for a quarterly filing.
  • Serious error: The model mis‑classified a £10 000 founder‑share injection as a Capital Account entry instead of the legally required Unpaid Shares (share capital). This mis‑classification has potential audit and filing implications, even though it did not affect the VAT total.
  • Common minor errors:
    • Zero‑rated vs. tax‑exempt confusion in 14 transactions. Both categories are VAT‑free, but they have distinct legal meanings. The model made this mistake consistently in January and February but corrected itself in March.
    • Split‑transaction VAT double‑counting in three Wise‑card transactions. In two cases the model applied VAT to both legs of a split payment; in the third case it correctly redistributed the VAT across the legs, though the scorer still marked it as an error.
  • What the model always got right:
    • Correct chart‑of‑accounts categorisation for every transaction except the share‑capital case.
    • Accurate attachment of the proper receipt to each transaction.
    • Robust disambiguation of ambiguous inputs (e.g., identical‑amount, same‑vendor entries, transfers, and disguised card purchases).

Community reactions and practical considerations

“The job performed by the humans was broader than what was requested of the model… humans also had to find the relevant invoices and reason about undocumented circumstances.” – Diogenesian

The HN discussion highlighted several themes:

  1. Scope of automation – Human accountants perform additional tasks (invoice retrieval, judgment calls) that were supplied to the model as explicit notes. Automating those steps remains an open challenge.
  2. Liability concerns – Commenters warned that while a model may be “nearly” as accurate, tax authorities demand perfect compliance. Errors, even if financially immaterial, could expose businesses to penalties or legal risk.
  3. Regulatory trust – Some users expressed reluctance to replace a regulated professional with an LLM, citing the lack of accountability and the need for a human to sign off on filings.
  4. Existing tooling – Several participants noted that many SaaS accounting platforms already generate deterministic VAT returns from structured invoice data, reducing the marginal benefit of an LLM in a fully digitised workflow.
  5. Future potential – Others were optimistic, pointing to open‑source efforts (e.g., beansync) and the rapid improvement of LLM reasoning as evidence that fully automated bookkeeping could soon be a commodity.

Implications for SMEs and the bookkeeping market

  • Cost reduction: At under $3 per quarter, GLM‑5.2 offers a cost advantage of > 99 % compared with traditional accounting firms.
  • Speed: The entire quarter’s VAT return was produced in just over an hour of wall‑time, far faster than the multi‑day turnaround typical of human accountants.
  • Accuracy ceiling: The benchmark demonstrates that LLMs can reach near‑human precision on deterministic accounting tasks, but edge‑case legal classifications (e.g., share‑capital treatment) still require human oversight.
  • Productisation path: The authors are building a SaaS front‑end (toot‑books.com) to expose this capability to UK startups. Success will hinge on integrating reliable data ingestion (email, bank APIs) and providing a clear liability framework for users.

Bottom line

GLM‑5.2 can generate a quarterly UK VAT return with sub‑penny deviation from a human‑prepared filing while costing a few dollars in compute. The model excels at deterministic classification and receipt attachment, but it still misclassifies legally significant items such as share‑capital entries and occasionally confuses zero‑rated versus exempt VAT categories. For SMEs, the technology promises dramatic cost and time savings, yet full adoption will require robust data pipelines, liability coverage, and a human‑in‑the‑loop for edge‑case judgments.

References

This article is based solely on the publicly available benchmark description and the accompanying Hacker News comments. No additional data or proprietary information was used.

Sources