Adversarial Training Methods for Semi-Supervised Text Classification

OpenAI researchers have introduced a method for applying adversarial training and virtual adversarial training (VAT) to text classification, enabling better regularization and the use of unlabeled data in semi-supervised settings. By shifting perturbations from the raw input level to the word embedding level, the researchers achieved state-of-the-art results on multiple benchmark tasks.

Adapting Adversarial Training for Sparse Text Inputs

Adversarial training serves as a regularization technique for supervised learning, while virtual adversarial training allows supervised algorithms to be extended into semi-supervised learning. However, traditional adversarial methods require small perturbations to be made to numerous entries of the input vector. This approach is ineffective for sparse, high-dimensional inputs like one-hot word representations, where small changes to a single entry cannot be meaningfully applied.

To solve this problem, the researchers applied perturbations to the word embeddings within a recurrent neural network (RNN) rather than to the original one-hot input. This shift allows the model to be regularized in a continuous space where small perturbations are mathematically viable and effective.

Performance and Capabilities

Applying adversarial perturbations to word embeddings has led to several key improvements in model performance:

  • State-of-the-Art Results: The proposed method achieved state-of-the-art performance on multiple benchmark tasks, both in purely supervised and semi-supervised settings.
  • Reduced Overfitting: Analysis during the training process indicates that the model is less prone to overfitting when using these adversarial methods.
  • Improved Embedding Quality: Visualizations and analysis demonstrate that the learned word embeddings have improved in quality compared to standard training methods.

Technical Implementation and Availability

The research was conducted by Takeru Miyato, Andrew M. Dai, and Ian Goodfellow. The implementation is based on a recurrent neural network architecture and the code for the method is available via the TensorFlow models research repository.

Sources