✷ The archive · 11 labs · 3,433 dispatches
The labs
No more opening a dozen official blogs every morning. First-hand releases from OpenAI, Anthropic, DeepMind and the rest, each with its substance pulled out.
Retrieval Augmented Generation with Hugging Face Transformers and Ray
Hugging Face has integrated Ray into the Retrieval Augmented Generation (RAG) model's document retrieval mechanism to achieve a 2x speedup in retrieval calls and improve distributed fine-tuning scalability.
Hugging Face PyTorch / XLA TPU Integration
Hugging Face has integrated PyTorch / XLA to enable PyTorch users to train and scale transformer models on Cloud TPUs using the existing Hugging Face Trainer interface.
Understanding the Capabilities, Limitations, and Societal Impact of Large Language Models
OpenAI summarizes a multidisciplinary workshop exploring the technical boundaries and societal risks associated with GPT-3 and other large language models.
Hugging Face Transformers v4.2.0 TensorFlow Performance and Serving Updates
Hugging Face Transformers v4.2.0 introduces significant computational performance gains for TensorFlow models and streamlined deployment via TensorFlow Serving using the SavedModel format.
OpenAI Scaling Kubernetes to 7,500 Nodes
OpenAI scaled a single Kubernetes cluster to 7,500 nodes to provide a simple, scalable infrastructure for machine learning research, overcoming challenges in networking, API server load, and monitoring.
Hugging Face Transformers ZeRO Integration via DeepSpeed and FairScale
Hugging Face Transformers v4.2.0 introduces experimental support for DeepSpeed and FairScale's ZeRO optimizations, enabling the training of larger models with higher batch sizes and reduced GPU memory requirements.
Hugging Face Accelerated Inference API Optimization
Hugging Face achieved a 100x speedup in transformer inference for its Accelerated Inference API by combining high-level library optimizations, Rust-based tokenization, and hardware-specific compilation.
CLIP: Connecting Text and Images
OpenAI introduced CLIP, a neural network that learns visual concepts from natural language supervision to enable zero-shot image classification across diverse datasets.
DALL·E: Creating images from text
OpenAI introduced DALL·E, a 12-billion parameter transformer model capable of generating diverse images from text descriptions by treating images and text as a single stream of tokens.
OpenAI Organizational Update December 2020
OpenAI announced the departure of VP of Research Dario Amodei and the appointment of Mira Murati as SVP of Research, Product, and Partnerships to better integrate safety and product development.
Leveraging Pre-trained Language Model Checkpoints for Encoder-Decoder Models – Hugging Face Blog Summary
Hugging Face’s blog post explains how to warm-start encoder-decoder models using pre‑trained BERT, RoBERTa, or GPT2 checkpoints, showing that this approach matches the performance of large pre‑trained seq2seq models while cutting training cost.
Porting fairseq WMT19 translation system to 🤗 Transformers
Hugging Face ported the fairseq WMT19 translation models (en‑ru, ru‑en, de‑en, en‑de) to the 🤗 Transformers library, allowing users to load and run these high‑quality translators with the standard Transformers API.
Hugging Face Transformers and Ray Tune Integration
Hugging Face Transformers 3.1 introduces an integration with Ray Tune, enabling users to easily implement advanced hyperparameter tuning algorithms like Population-Based Training and Bayesian Optimization.
Transformer-based Encoder-Decoder Models Hugging Face Blog Post 2020
Hugging Face’s 2020 blog post explains the transformer-based encoder-decoder architecture, detailing how it maps input sequences to variable-length outputs via encoder and decoder stacks, self-attention, cross-attention, and autoregressive generation, and shows how to use it with the 🤗Transformers library.
OpenAI Licenses GPT-3 Technology to Microsoft
OpenAI has licensed its GPT-3 language model technology to Microsoft for integration into Microsoft products and services, while maintaining independent API access for third-party developers.
Hugging Face pytorch_block_sparse Release
Hugging Face has released pytorch_block_sparse, a library providing BlockSparseLinear modules to create smaller and faster language models by reducing memory consumption and improving computation efficiency over standard PyTorch sparse matrices.
Generative Language Modeling for Automated Theorem Proving
OpenAI introduces GPT-f, a transformer-based language model designed for the Metamath formalization language that successfully contributed new proofs to the main Metamath library.
Learning to Summarize with Human Feedback
OpenAI demonstrates that reinforcement learning from human feedback (RLHF) allows smaller language models to outperform significantly larger models trained only via supervised learning in text summarization.
OpenAI Scholars 2020 Final Projects
OpenAI announced the final projects of the 2020 Scholars program, showcasing research in neural network interpretability, reinforcement learning, semantic parsing, and medical AI.
Reformer: Pushing the Limits of Language Modeling with Memory-Efficient Transformers
The Reformer model, introduced by Hugging Face in July 2020, enables training on sequences up to half a million tokens using less than 8 GB of RAM by combining LSH self‑attention, local self‑attention, chunked feed‑forward layers, reversible residuals, and axial positional encodings.
OpenAI Procgen and MineRL Competitions
OpenAI is co-organizing two NeurIPS 2020 competitions using the Procgen Benchmark and MineRL to advance reinforcement learning sample efficiency and generalization.
Image GPT
OpenAI's Image GPT (iGPT) demonstrates that a transformer model trained on pixel sequences can generate coherent images and learn competitive unsupervised visual features without domain-specific architectural priors.
OpenAI API Release
OpenAI has released a general-purpose text-in, text-out API providing access to models from the GPT-3 family to enable the development of diverse AI-powered applications.
OpenAI GPT-3: Language Models as Few-Shot Learners
OpenAI introduced GPT-3, a 175 billion parameter autoregressive language model that demonstrates strong few-shot performance across diverse NLP tasks without requiring task-specific fine-tuning.
OpenAI AI and Efficiency Analysis
OpenAI's analysis reveals that algorithmic progress has reduced the compute needed to train neural networks to the same performance level by a factor of 2 every 16 months since 2012.
OpenAI Jukebox
OpenAI Jukebox is a neural network capable of generating raw audio music, including rudimentary singing, conditioned on genre, artist, and lyrics.
OpenAI: Improving Verifiability in AI Development
OpenAI and a multi-institutional coalition propose ten mechanisms to help stakeholders verify that AI systems adhere to stated ethics principles and safety standards to prevent competitive corner-cutting.
OpenAI Microscope: A Tool for Neural Network Interpretability
OpenAI Microscope is a visualization tool that systematically visualizes every neuron in several common vision models to accelerate the research into reverse-engineering neural networks.
Hugging Face Text Generation Decoding Methods Guide
Hugging Face provides a comprehensive overview of auto-regressive decoding strategies, including Greedy Search, Beam Search, and Sampling (Top-K and Top-p), to optimize open-ended language generation in Transformers.
Training a Language Model from Scratch with Transformers and Tokenizers
Hugging Face provides a comprehensive guide and demonstration on training a new language model from scratch using the Transformers and Tokenizers libraries, featuring the creation of EsperBERTo for the Esperanto language.
OpenAI Standardizes on PyTorch
OpenAI has standardized its deep learning framework on PyTorch to increase research productivity and reduce iteration times for generative modeling.
OpenAI Scaling Laws for Neural Language Models
OpenAI researchers discovered that language model performance follows predictable power-law relationships with model size, dataset size, and compute, enabling the optimal allocation of training resources.
OpenAI Five: Dota 2 Reinforcement Learning
OpenAI Five is the first AI system to defeat world champions in an esports game, demonstrating that large-scale self-play reinforcement learning can achieve superhuman performance in complex, continuous state-action spaces.
OpenAI Deep Double Descent
OpenAI researchers demonstrate that CNNs, ResNets, and transformers exhibit a double descent phenomenon where performance improves, degrades, and then improves again as model size, data size, or training time increases.
OpenAI Procgen Benchmark Release
OpenAI has released the Procgen Benchmark, a suite of 16 procedurally-generated environments designed to measure reinforcement learning agents' ability to generalize skills to unseen levels.
OpenAI Benchmarking Safe Exploration in Deep Reinforcement Learning
OpenAI introduces the Safety Gym benchmark suite to standardize constrained reinforcement learning as a primary formalism for safe exploration in high-dimensional continuous control environments.
OpenAI Safety Gym Release
OpenAI has released Safety Gym, a suite of environments and tools designed to measure and improve the ability of reinforcement learning agents to respect safety constraints during training.
GPT-2 1.5B release notes
OpenAI has released the largest version of GPT-2 with 1.5 billion parameters, completing its staged release process to provide a case study for responsible AI publication.
OpenAI Solving Rubik's Cube with a Robot Hand
OpenAI has trained a pair of neural networks to solve a Rubik's Cube using a human-like robot hand, utilizing a new technique called Automatic Domain Randomization (ADR) to transfer simulation-learned skills to the physical world.
OpenAI Scholars 2020 Program Applications
OpenAI has opened applications for its third OpenAI Scholars class, a four-month program providing stipends and mentorship to eight individuals from underrepresented groups to study deep learning.
Fine-tuning GPT-2 from human preferences
OpenAI fine-tuned the 774M parameter GPT-2 model using human feedback to improve stylistic text continuation and summarization, discovering that human labelers often prefer accurate copying over novel synthesis.
OpenAI Emergent Tool Use from Multi-Agent Interaction
OpenAI researchers discovered that agents playing a simulated game of hide-and-seek developed six distinct levels of complex tool use and strategies through multi-agent co-adaptation and self-play.
OpenAI Testing Robustness Against Unforeseen Adversaries
OpenAI has introduced a new metric, Unforeseen Attack Robustness (UAR), to evaluate how well neural network classifiers defend against adversarial attacks not encountered during training.
GPT-2 6-month follow-up
OpenAI released the 774M parameter GPT-2 model and shared lessons on the risks of synthetic text, the difficulty of detection, and the necessity of staged release strategies for large language models.
OpenAI Learning Day: Cultivating Cross-Functional Expertise
OpenAI implements Learning Day, a weekly dedicated self-study day every Thursday to foster cross-functional technical growth and prevent professional stagnation.
Microsoft and OpenAI Partnership for AGI Development
Microsoft invested $1 billion in OpenAI to co-develop a hardware and software platform on Azure designed to scale toward the creation of beneficial artificial general intelligence (AGI).
OpenAI Framework for AI Safety Cooperation
OpenAI proposes four strategies to prevent collective action problems in AI development by promoting cooperation on safety norms and standards across the industry.
OpenAI Robotics Symposium 2019
OpenAI hosted its first Robotics Symposium on April 27, 2019, bringing together experts from robotics and machine learning to discuss the development of robots that learn.
OpenAI Scholars 2019 Final Projects
OpenAI released a summary of final projects from the 2019 OpenAI Scholars program, showcasing diverse applications of deep learning, reinforcement learning, and NLP across various domains.
OpenAI Fellows Fall 2018 Final Projects
OpenAI announced the completion of its second class of OpenAI Fellows, with all six participants transitioning to full-time technical staff after a six-month apprenticeship.