Large Language Models: A New Moore's Law?

The trend of exponentially increasing large language model (LLM) sizes is leading to diminishing returns, prohibitive costs, and significant environmental impacts. While massive models like the Megatron-Turing NLG 530B demonstrate engineering prowess, the path toward trillion-parameter models is often impractical for real-world business applications and unsustainable for the planet.

The Costs of Mega-Model Scaling

Scaling models to hundreds of billions of parameters creates extreme financial and environmental barriers to entry. The pursuit of "mega-models" relies on brute-force engineering rather than algorithmic efficiency.

Financial Prohibitiveness

Training a model of the scale of Megatron-Turing NLG 530B requires massive infrastructure. Using hundreds of DGX A100 multi-GPU servers—costing approximately $199,000 each—alongside networking and hosting costs, replicating such an experiment could cost close to $100 million. Very few organizations have business use cases that justify such an investment.

Environmental Impact

Training deep learning models on GPUs is energy-intensive. A single DGX server can consume up to 6.5 kilowatts. The carbon footprint of these models is substantial; for context, a 2019 University of Massachusetts study noted that training BERT on a GPU is roughly equivalent to a trans-American flight. BERT-Large has only 340 million parameters, suggesting an even more massive footprint for models in the 500-billion parameter range.

Pragmatic Alternatives to Massive Models

Instead of chasing model size, developers should focus on efficient, actionable techniques to build high-quality machine learning solutions.

Leveraging Pretrained and Smaller Models

Most use cases do not require custom model architectures. The most efficient workflow is to identify a pretrained model for the specific task (e.g., text summarization) and test it on target data. If accuracy is insufficient, the model should be fine-tuned.

Furthermore, practitioners should select the smallest model that meets their accuracy requirements to ensure faster prediction and lower hardware overhead. Examples of efficiency-driven research include:

  • SqueezeNet: Achieved a 50x reduction in size compared to AlexNet while maintaining or exceeding accuracy.
  • DistilBERT: Retains 97% of BERT's language understanding while being 40% smaller and 60% faster.
  • T0 (Big Science project): Outperforms GPT-3 on many tasks while being 16x smaller.

Fine-Tuning vs. Training from Scratch

Fine-tuning a pretrained model on a specific dataset for a few epochs is significantly more efficient than training from scratch. This approach of transfer learning reduces the need for massive data collection, accelerates iteration cycles, and lowers production resource requirements.

Infrastructure and Optimization Strategies

To maximize efficiency and sustainability, organizations should utilize optimized infrastructure and specialized software tools.

Cloud-Based Infrastructure

Cloud-based infrastructure is generally more energy and carbon efficient than on-premises alternatives. Providers like AWS, Azure, and Google Cloud offer managed services (such as Amazon SageMaker) that provide flexibility and a pay-as-you-go model, reducing the waste associated with underutilized hardware.

Model Optimization Techniques

Optimizing models for size and speed involves several complex technical strategies:

  • Specialized Hardware: Utilizing Graphcore, Habana, Google TPU, or AWS Inferentia to speed up training and inference.
  • Pruning: Removing model parameters that have little to no impact on the predicted outcome.
  • Fusion: Merging model layers, such as combining convolution and activation layers.
  • Quantization: Storing model parameters in smaller values (e.g., 8-bit instead of 32-bit).

Tools like the open-source Optimum library and Infinity (a containerized solution for 1-millisecond latency) are designed to automate these optimization processes.

Sources