Making ML-powered web games with Transformers.js

Hugging Face has detailed the development of Doodle Dash, a real-time machine learning (ML) powered web game that operates entirely within the user's browser. By leveraging Transformers.js, the game eliminates server latency, allowing the model to make over 60 predictions per second as the user draws.

Model Training and Architecture

The core of Doodle Dash is a sketch detection model trained on a subset of Google's "Quick, Draw!" dataset, which contains over 5 million drawings across 345 categories.

Model Selection

The developers finetuned apple/mobilevit-small, a lightweight Vision Transformer (ViT) pre-trained on ImageNet-1k. This model was chosen for its mobile-friendly architecture and small footprint, consisting of only 5.6 million parameters and a file size of approximately 20 MB, making it ideal for in-browser execution.

Finetuning Process

The finetuning workflow involved the following technical steps:

  1. Data Loading: Importing the "Quick, Draw!" dataset subset.
  2. Preprocessing: Using MobileViTImageProcessor to transform the data.
  3. Configuration: Defining the collate function and evaluation metrics.
  4. Model Initialization: Loading the pre-trained MobileViTForImageClassification model.
  5. Training: Utilizing the Trainer and TrainingArguments helper classes.
  6. Evaluation: Using the 🤗 Evaluate library to verify performance.

Browser Deployment with Transformers.js

Transformers.js is a JavaScript library that enables the execution of 🤗 Transformers models directly in the browser without a backend server. It provides an API functionally equivalent to the Python library.

Model Conversion to ONNX

Because Transformers.js utilizes ONNX Runtime for execution, PyTorch models must be converted to the ONNX format. This is achieved using the 🤗 Optimum library via a conversion script provided in the Transformers.js repository:

python -m scripts.convert --model_id <model_id>

Technical Implementation

To prevent the computationally intensive inference process from blocking the main UI thread (which handles rendering and user input), the game implements the Web Workers API. The inference logic is isolated in a separate worker thread (worker.js), which initializes the image-classification pipeline and processes grayscale images to return predictions.

Game Design and Optimization

Real-Time Inference Loop

Unlike the original "Quick, Draw!" game, which makes predictions every few seconds, Doodle Dash utilizes the high-frequency performance of in-browser inference to create a faster game loop:

  • Objective: Players attempt to draw as many doodles as possible within 60 seconds.
  • Mechanics: The canvas clears immediately upon a correct prediction, prompting a new word.
  • Penalty: Skipping a word costs the player 3 seconds of remaining time.
  • Score Adjustment: To prevent the model from simply crossing off labels, the game decreases scores for the first n incorrect labels, with n increasing over time.

Dataset Refinement

To maintain game quality despite the small model size (~20MB), the developers filtered the original 345 classes. Labels were removed if they were:

  • Too similar to other labels (e.g., "barn" vs. "house").
  • Too difficult to understand or draw in sufficient detail (e.g., "animal migration" or "brain").
  • Ambiguous (e.g., "bat").

This filtering process resulted in a final set of over 300 distinct classes.

Sources