TAPEX: Efficient Table Pre-training via Neural SQL Execution

TL;DR

TAPEX (Table Pre-training via Execution) is a pre-training method that teaches language models to act as neural SQL executors using synthetic data. This approach closes the gap between general pre-training and downstream table tasks, achieving state-of-the-art performance while requiring significantly less data than traditional table pre-training methods.

Neural SQL Execution as a Pre-training Task

TAPEX replaces traditional natural language pre-training with a task focused on learning to execute SQL queries. Instead of relying on large-scale, often noisy, textual data from the web, TAPEX synthesizes a non-natural pre-training corpus by systematically sampling executable SQL queries and their corresponding execution outputs over tables.

The Pre-training Process

To create the training data, TAPEX follows these steps:

  1. Table Selection: A table is retrieved from the web.
  2. Query Sampling: An executable SQL query is sampled (e.g., SELECT City WHERE Country = France ORDER BY Year ASC LIMIT 1).
  3. Execution: An off-the-shelf SQL executor (such as MySQL) generates the ground-truth result (e.g., Paris).
  4. Model Training: A language model (such as BART) is trained to take the concatenation of the SQL query and the flattened table as input and produce the execution result as output.

By using programs like SQL rather than natural language, TAPEX ensures the diversity and scale of the pre-training corpus are systematically guaranteed and high-quality.

Fine-tuning and Downstream Application

After pre-training as a neural SQL executor, the model is fine-tuned for real-world applications. During this phase, the model is fed a concatenation of a natural language question and a flattened table, and it is trained to produce the answer labeled by human annotators.

This transition from SQL execution to natural language question answering is effective because the reasoning paths required to resolve a SQL query and a natural language question with similar intent are nearly identical, though SQL is more rigid.

Performance Benchmarks and Results

TAPEX achieves new state-of-the-art (SOTA) results across four major benchmark datasets, outperforming previous table pre-training approaches by significant margins:

  • WikiTableQuestions: 57.5% denotation accuracy (+4.8% over SOTA, +19.5% over BART).
  • SQA: 74.5% denotation accuracy (+3.5% over SOTA, +15.9% over BART).
  • TabFact: 84.2% accuracy (+3.2% over SOTA, +3.0% over BART).
  • WikiSQL (Weak): 89.6% denotation accuracy (+2.3% over SOTA, +3.8% over BART).

Efficiency Comparison with Previous Methods

TAPEX demonstrates a massive increase in pre-training efficiency compared to earlier models like TAPAS (Google Research) and TaBERT (Meta AI). While previous models relied on general-purpose pre-training tasks like language modeling, TAPEX uses a domain-adaptive task (SQL execution).

Experimental data shows that TAPEX can achieve a 2% performance improvement over TaBERT while using only 2% of the pre-training corpus, representing a speedup of nearly 50 times.

Key Technical Takeaways

For those performing efficient continual pre-training, the success of TAPEX suggests two primary strategies:

  1. Prioritize Synthetic Accuracy: Synthesize an accurate, small corpus rather than mining a large, noisy corpus from the internet.
  2. Simulate Domain Skills via Programs: Use programs to simulate domain-adaptive skills instead of relying on general-purpose language modeling via natural language sentences.

Sources