Building self-improving tax agents with Codex

OpenAI and Thrive Holdings co-developed Tax AI for Crete accountants, creating a system that uses Codex to transform production failures into structured signals for autonomous improvement. This approach allows the agent to evolve from handling simple forms to complex tax filings without requiring manual engineering for every edge case.

Measurable Performance Gains

Tax AI significantly reduces the manual burden of tax preparation for 1040 and 1041 returns, saving practitioners approximately one-third of their preparation time and increasing throughput by 50%.

• Accuracy: The system drafts returns with up to 97% accuracy.
• Completion Rates: At launch, only 25% of returns reached a 75% correct field completion rate. Within six weeks, this increased to 86%.
• Complexity Scaling: The system initially handled basic documents like W-2s and 1099s before expanding into complex K-1s and schedules, with each new capability providing greater time savings per return.

The Three-Part Self-Improvement Loop

To move beyond manual prompt adjustments and engineering fixes, the team implemented a three-pillar architecture that enables the agent to improve itself based on real-world usage.

1. Expert Practitioner Feedback

Practitioners steer the learning process through their daily work. By correcting system errors during the review process, accountants provide the ground truth that identifies which errors are critical and which parts of the workflow require the most attention.

2. Production Traces as Evidence

Tax AI captures the full path of a transaction—from the original source material and extracted fields (with citations) to the final downstream submission and expert correction. This structured history allows the team to distinguish between true extraction misses, mapping problems, or expected workflow noise.

3. Codex-Driven Iteration

Once production issues are identified and grouped into actionable findings, they are converted into targeted evaluation (eval) sets. Codex then operates within a bounded engineering loop to:

• Investigate: Analyze source packages, extraction schemas, and code paths to find the root cause.
• Implement: Update extraction schemas, improve source selection, or refine the tax-engine mapper.
• Validate: Rerun targeted evals and broader regression suites to propose a candidate pull request for human engineering review.

Case Study: Rental Property Extraction (Schedule E)

Extracting rental property income is complex due to messy source materials such as handwritten notes and spreadsheets. The self-improvement loop handles this as follows:

1. Correction: A practitioner modifies a value in the filed return that differs from the agent's prediction.
2. Trace Analysis: The system compares the output with the filed return to produce field-level review rows. Repeated patterns (e.g., consistently missing "fair rental days") are grouped into eval targets.
3. Codex Resolution: Codex receives the targeted eval set and investigates the product scaffold. It may extend the extraction schema or update the mapper to resolve the specific failure, then validates the fix against the regression suite.

Implementation Framework for Codex

Building this loop requires a bounded task environment that separates the writable worktree from read-only production context.

• Writable Worktree: Contains the scoped product surface Codex can modify, the targeted/regression evals, and reusable skills/documentation.
• Read-Only Context: Provides the production trace, source documents, the agent's prediction, the finalized return, and tax-engine field documentation.

This structure ensures Codex can investigate failures without mutating the underlying evidence, while engineers remain responsible for overall architecture and final shipping decisions.

Broader Implications and Domain Expansion

The patterns established with Tax AI—using production artifacts and traces to drive agent improvement—are being applied to other domains within Thrive Holdings, including bookkeeping, audit, and IT help desk automation. The project demonstrates that the most effective agents are those steered by human experts to become more capable and trusted over time.