Dynamic Adversarial Data Collection (DADC) for MNIST – Hugging Face tutorial

TL;DR

Hugging Face released a practical guide that demonstrates Dynamic Adversarial Data Collection (DADC) on the MNIST handwritten digit task, enabling users to collect adversarial examples via a Gradio 🤗 Space, flag them, and iteratively retrain the model for greater robustness.

What is Dynamic Adversarial Data Collection?

Dynamic Adversarial Data Collection (DADC) replaces static benchmarks with a loop where humans create inputs that deliberately fool a state‑of‑the‑art model. The loop provides two benefits: it reveals the true robustness of the model and generates a growing, human‑aligned dataset that can be used to train a stronger model. Repeating the fool‑and‑retrain cycle over multiple rounds yields a model that aligns better with human expectations.

Configuring the MNIST Model

The tutorial starts by defining a simple convolutional neural network for MNIST:

class MNIST_Model(nn.Module):
    def __init__(self):
        super(MNIST_Model, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.conv2_drop = nn.Dropout2d()
        self.fc1 = nn.Linear(320, 50)
        self.fc2 = nn.Linear(50, 10)

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = x.view(-1, 320)
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return F.log_softmax(x)

After defining the architecture, the model is trained on the standard MNIST train/dev split.

Interacting with the Model via 🤗 Spaces

Once the baseline model is trained (the example reaches 89 % test accuracy after 20 epochs), the guide shows how to expose it through a Gradio 🤗 Space. Users draw digits on a canvas; the model returns its prediction. The Space URL is https://huggingface.co/spaces/chrisjay/simple-mnist-classification. This interactive demo is the entry point for collecting adversarial inputs.

Flagging Adversarial Examples

When a user succeeds in fooling the model, the example is flagged:

  1. The adversarial image is saved to a dataset.
  2. After a threshold number of flagged samples is reached, the model is retrained on the expanded dataset.
  3. Steps 1‑2 are repeated across multiple rounds. The tutorial provides a custom flag function (see the full source at .../mnist-adversarial/blob/main/app.py#L314). Gradio also offers a built‑in flagging callback, documented at https://gradio.app/using_flagging/.

End‑to‑End Demo

All three components—model definition, interactive Space, and flagging logic—are combined into a single demo called MNIST Adversarial (https://huggingface.co/spaces/chrisjay/mnist-adversarial). Users can test the system directly, generate adversarial samples, and observe how the model improves after retraining.

Implications and Community Findings

DADC addresses the limitations of static benchmarks, such as saturation and hidden biases, by continuously injecting human‑generated edge cases. The blog cites research by Eric Wallace et al. (2022) showing that while non‑adversarial data collection may yield short‑term gains, DADC leads to the highest long‑term accuracy on natural language inference tasks. The same principle applies to vision tasks like MNIST, where diverse handwriting styles are under‑represented in static test sets.

Conclusion

Dynamic Adversarial Data Collection enables the creation of non‑saturating, human‑aligned datasets and improves model robustness through an iterative fool‑and‑retrain loop. Hugging Face’s tutorial demonstrates that building a full DADC pipeline is straightforward with 🤗 Spaces and Gradio, lowering the barrier for researchers and practitioners to adopt this approach.

Sources