2023 Year of Open LLMs Review

TL;DR

2023 marked a rapid expansion of open‑source large language models (LLMs), with many new releases ranging from 3B to 70B parameters, extensive fine‑tuning methods, and tooling that made LLMs far more accessible to researchers and developers.

🍜 How a pretrained LLM is built

  • Architecture – Most top‑performing open LLMs use a decoder‑only Transformer (the classic "transformer" design from the 2017 paper). Variations include ALiBi, RoPE, RMSNorm, SwiGLU, and attention tweaks such as Flash‑Attention, GQA, or sliding windows.
  • Training dataset – Textual data (natural language, code, tables, equations) is tokenized into sub‑word units; vocabularies typically span 32k–200k tokens. Modern datasets contain hundreds of billions to several trillion tokens.
  • Tokenizer – Converts raw text into numeric token IDs; the token count of a dataset is the primary scale metric.
  • Training hyper‑parameters – Define learning rates, batch sizes, and other optimizer settings that control how weights are updated.
  • Compute – Large‑scale training requires massive GPU/TPU clusters and careful monitoring.
  • Weights – The resulting learned parameters are released as the model checkpoint, enabling inference and downstream fine‑tuning.
  • Fine‑tuning – Open weights allow inexpensive adaptation to specific tasks, often with far less compute than training from scratch.

🗝️ 2022: From size‑driven to data‑driven scaling

  • BLOOM (BigScience, 176B params, 350B tokens, multilingual) – Open, fully documented data pipeline.
  • OPT (Meta, 175B params, 180B tokens) – GPT‑3‑level performance with compute‑efficient tricks.
  • GLM‑130B (Tsinghua/Zhipu.AI, 130B params, 400B tokens) – Comparable to GPT‑3, with DeepNorm and rotary embeddings.
  • Smaller open models – Galactica (up to 120B, scientific text) and GPT‑NeoX‑20B (20B, 500B tokens) demonstrated feasibility of fully open stacks.
  • Shift in scaling law – DeepMind’s Chinchilla paper (70B params, 1.4T tokens) showed that, for a fixed compute budget, smaller models trained on more data outperform larger, data‑starved models. This insight sparked a community‑wide move toward “more data, smaller models.”

🌊 2023: A wave of open releases

The rise of small LLMs (3B–70B)

  • LLaMA (Meta, Feb) – 65B‑param model trained on 1.4T tokens; 6B/13B variants trained on 1T tokens; non‑commercial license.
  • Pythia (EleutherAI, Apr) – Scaling suite of publicly trained models on fully disclosed data.
  • MPT (MosaicML, May) – 7B and 30B models, commercial‑friendly license, trained on 1T tokens of English + code.
  • Falcon (TII‑UAE, Jun) – 7B/30B (later 180B) models, 1–1.5T tokens, detailed technical report.
  • StableLM (StabilityAI, Apr & Aug) – 3B/7B base models (1.5T tokens) and later v2 series with mixed data sources.
  • X‑Gen (Salesforce, Jun) – 7B model, 1.5T tokens, staged data scheduling.
  • LLaMA‑2 (Meta, Jul) – 7–70B models, 2T tokens, permissive community license, extensive RLHF alignment.
  • Mistral‑7B (Mistral.AI, Sep) – Trained on undisclosed web data; later Mixtral‑8×7B introduced MoE routing.
  • Qwen (Alibaba, Sep) – 7–70B models, 2.4T tokens, bilingual English‑Chinese focus.
  • Yi (01‑AI, Nov) – 6–34B models, 3T tokens, strong leaderboard performance.
  • DeciLM, SOLAR, and others (Deci, Upstage, etc.) – Continued incremental improvements on the 7–10B scale.

All these releases share:

  • Decoder‑only Transformer core.
  • Varied architectural tweaks (ALiBi, RoPE, RMSNorm, SwiGLU, Flash‑Attention, GQA, sliding windows).
  • Open weights under a range of licenses, from non‑commercial to fully permissive.
  • Emphasis on smaller models trained on massive token counts, echoing the Chinchilla insight.

Dialog models everywhere

  • Chat‑based fine‑tuning – Supervised training on multi‑turn dialogue data.
  • Instruction fine‑tuning (IFT) – Uses instruction‑response pairs, often synthetic (e.g., GPT‑4‑generated).
  • RLHF – Human‑ranked responses train a preference model, which then guides reinforcement learning; costly but improves safety.
  • RLAIF – Replaces human rankers with a high‑quality LLM for preference scoring.
  • Direct Preference Optimization (DPO) – Updates the model directly from ranking data, bypassing a separate preference model.

Most 2023 releases shipped both a base checkpoint and a chat‑tuned variant (e.g., LLaMA‑2‑Chat, Falcon‑Instruct, MPT‑Chat, Qwen‑Chat, Yi‑Chat). LLaMA‑2’s safety‑focused alignment was especially notable.

Community activity

  • Massive proliferation of instruction and chat datasets: WebGPT, HH‑RLHF, P3, FLAN, Self‑Instruct, HC3, Alpaca, OIG, Vicuna, UltraChat, Open‑Orca, UltraFeedback, Zephyr, OpenHermes, Starling, Nectar, etc.
  • Community fine‑tunes (Alpaca, Koala, Dolly, UltraLLaMA, Hermes, Capybara, OpenChat, etc.) built on top of the base models, often leveraging RLHF, DPO, or RLAIF.
  • The ecosystem created a virtuous loop: high‑quality base models enable new datasets, which in turn produce stronger fine‑tuned models.

Democratizing access

Model merging (extreme customization)

  • Merging averages or weighted‑averages the weights of multiple compatible models, sometimes applying interference‑aware techniques (e.g., ties merging). This yields hybrids like llama2‑zephyr‑orca‑ultra that combine strengths of several checkpoints.

Parameter‑efficient fine‑tuning (PEFT)

  • Freeze the base model and train lightweight adapters (e.g., LoRA, IA³). Only the adapter weights need to be stored and shared, dramatically reducing storage and compute requirements.

Quantization (running everywhere)

  • Reducing numeric precision (float32 → float16 → int8 → 4‑bit) cuts memory footprints: a 30B model drops from ~66 GB (float16) to ~33 GB (8‑bit) or ~16 GB (4‑bit) with minimal performance loss on large models.
  • Popular toolkits: bitsandbytes, GPTQ, AWQ; community converters (e.g., TheBloke) make quantized checkpoints widely available.

What’s next?

  • Mixture‑of‑Experts (MoE) – Mixtral‑8×7B routes each token through two of eight sub‑models, scaling parameter count without proportional compute.
  • State‑space models – Mamba and Striped Hyena introduce SSM architectures that may outperform Transformers on certain tasks; research is still early.

Takeaways

  • Open LLM releases exploded in 2023, enabling rapid experimentation across academia, industry, and hobbyist communities.
  • The trend shifted from “bigger is better” to “smaller, data‑rich models” after the Chinchilla scaling insight.
  • New fine‑tuning paradigms (RLHF, DPO, adapters, merging) dramatically expanded personalization options.
  • Quantization and PEFT lowered hardware barriers, allowing models to run on consumer‑grade GPUs.
  • Emerging actors from China (Qwen, Yi) and novel architectures (MoE, SSM) suggest the open‑source LLM landscape will remain highly competitive.

Sources