✷ The archive · 11 labs · 3,064 dispatches
The labs
No more opening a dozen official blogs every morning. First-hand releases from OpenAI, Anthropic, DeepMind and the rest, each with its substance pulled out.
Hosting Models and Datasets on Hugging Face Spaces using Streamlit
Hugging Face Spaces integrates with Streamlit to allow users to quickly build and host interactive demos for machine learning models and data visualizations.
Hugging Face Summer 2021 Update
Hugging Face announced a series of Hub enhancements, including the Spaces Beta for ML demo hosting, TensorBoard integration, and the launch of the Optimum library for hardware acceleration.
OpenAI Summarizing Books with Human Feedback (2021)
OpenAI introduced a technique that combines reinforcement learning from human feedback with recursive task decomposition to generate summaries of entire books, addressing the alignment challenge of overseeing AI on difficult-to-evaluate tasks.
Hugging Face Optimum Release
Hugging Face has introduced Optimum, an open-source toolkit designed to optimize Transformer models for production performance across various hardware platforms.
Hugging Face and Graphcore Partnership for IPU-Optimized Transformers
Hugging Face and Graphcore have partnered to integrate Intelligence Processing Units (IPUs) into the Hugging Face ecosystem via the Hardware Partner Program and the Optimum library to accelerate Transformer model deployment.
TruthfulQA: Measuring how models mimic human falsehoods
OpenAI introduces TruthfulQA, a benchmark designed to measure whether language models mimic common human misconceptions and falsehoods across 38 categories.
Helen Toner joins OpenAI board of directors
OpenAI appointed Helen Toner, an expert in AI policy and global strategy, to its board of directors on September 8, 2021, to strengthen its commitment to safe and responsible AI deployment.
OpenAI Codex: A GPT-3 Descendant for Code Generation
OpenAI Codex is a GPT-3 descendant trained on both natural language and billions of lines of public source code, enabling the generation of working code from English commands.
OpenAI Triton 1.0 Release
OpenAI has released Triton 1.0, an open-source Python-like language and compiler that allows researchers to write highly efficient GPU kernels without deep CUDA expertise.
Hugging Face DeDLOC: Collaborative Training of Language Models over the Internet
Hugging Face introduces DeDLOC, a distributed training method that enables volunteers to collaboratively pretrain large language models over the internet by adapting to varying network and hardware constraints.
spaCy Integration with Hugging Face Hub
Hugging Face has integrated spaCy into the Hugging Face Hub, allowing users to share, discover, and deploy spaCy pipelines via a unified platform.
Deploy Hugging Face models easily with Amazon SageMaker
Hugging Face and Amazon SageMaker have introduced new Inference Deep Learning Containers (DLCs) and an Inference Toolkit to simplify the deployment of Transformer models to production-ready endpoints.
Evaluating Large Language Models Trained on Code: OpenAI Codex
OpenAI introduces Codex, a GPT model fine-tuned on GitHub code, which solves 28.8% of HumanEval problems on a single attempt and up to 70.2% with repeated sampling.
Sentence Transformers Integration in the Hugging Face Hub
Hugging Face has integrated Sentence Transformers into the Hub, providing over 90 pretrained models for 100+ languages and new interactive widgets for feature extraction and sentence similarity.
OpenAI Research: Improving Language Model Behavior via Curated Dataset Fine-Tuning
OpenAI researchers found that fine-tuning GPT-3 on a small, curated dataset of fewer than 100 examples can significantly improve model adherence to specific behavioral values without compromising downstream performance.
Few-Shot Learning with GPT-Neo and Hugging Face Accelerated Inference API
Hugging Face explores the application of few-shot learning using the open-source GPT-Neo model and the Accelerated Inference API to enable task generalization without extensive labeled data.
Anthropic Series A Funding for Reliable General AI Systems
Anthropic has raised $124 million in Series A funding to develop large-scale AI systems that are steerable, interpretable, and robust.
Using & Mixing Hugging Face Models with Gradio 2.0
Gradio 2.0 enables machine learning developers to load and deploy Hugging Face models as GUIs with a single line of code, supporting both parallel and serial model composition.
OpenAI Scholars 2021: Final Projects
OpenAI announced the final projects of the 2021 Scholars program, featuring research on scaling laws, reward modeling, and reinforcement learning from a diverse group of researchers.
Will Hurd Joins OpenAI Board of Directors
OpenAI has appointed former U.S. Congressman Will Hurd to its board of directors to integrate public policy expertise with technical AI development.
Scaling-up BERT Inference on CPU (Part 1)
Hugging Face explores hardware-level optimizations for BERT inference on modern CPUs, demonstrating that throughput can be scaled linearly by using multiple independent model instances bound to specific physical cores via NUMA-aware affinity.
Hugging Face Accelerate Library Release
Hugging Face has released Accelerate, a PyTorch library that allows users to run raw training scripts on any device configuration, including multi-GPU and TPU, without rewriting boilerplate code.
Distributed Training of BART and T5 for Summarization via Hugging Face and Amazon SageMaker
Hugging Face and Amazon SageMaker have integrated to provide optimized Deep Learning Containers and a dedicated HuggingFace estimator to simplify distributed training of Transformers models like BART and T5.
Understanding BigBird's Block Sparse Attention
BigBird introduces block sparse attention to reduce the computational complexity of Transformers from quadratic to linear, enabling the processing of sequences up to 4096 tokens.
GPT-3 API Ecosystem and Application Growth
OpenAI reports that over 300 applications and tens of thousands of developers are using GPT-3 to generate an average of 4.5 billion words per day across diverse industries.
Amazon SageMaker and Hugging Face Partnership
Hugging Face and Amazon have partnered to integrate Hugging Face Transformers into Amazon SageMaker via dedicated Deep Learning Containers (DLCs) and a Python SDK extension to accelerate NLP model training and deployment.
Deploying a Hugging Face Transformers Sentiment Analysis Pipeline on Google Cloud Run
A community member demonstrated how to serve a Hugging Face sentiment‑analysis pipeline on Google Cloud Run using a DistilBERT model, achieving sub‑5‑second latency with minimal monthly cost.
Fine-Tuning Wav2Vec2 for English ASR with Hugging Face Transformers
Hugging Face provides a detailed guide on fine-tuning the Wav2Vec2 pretrained speech model for English Automatic Speech Recognition (ASR) using Connectionist Temporal Classification (CTC) loss.
Hugging Face Reads: Long-range Transformers
Hugging Face analyzes four key architectures—Longformer, Compressive Transformer, Linformer, and Performer—designed to reduce the quadratic memory and time complexity of standard Transformer self-attention to linear complexity.
Multimodal Neurons in CLIP
OpenAI discovered neurons in CLIP that respond to the same concept whether presented literally, symbolically, or conceptually, mirroring multimodal neurons found in the human brain.
Hugging Face: Simple Considerations for Building Neural Networks
Hugging Face provides a framework for building and debugging neural networks by prioritizing data analysis, simple baselines, and rigorous implementation checks over blind hyperparameter tuning.
Retrieval Augmented Generation with Hugging Face Transformers and Ray
Hugging Face has integrated Ray into the Retrieval Augmented Generation (RAG) model's document retrieval mechanism to achieve a 2x speedup in retrieval calls and improve distributed fine-tuning scalability.
Hugging Face PyTorch / XLA TPU Integration
Hugging Face has integrated PyTorch / XLA to enable PyTorch users to train and scale transformer models on Cloud TPUs using the existing Hugging Face Trainer interface.
Understanding the Capabilities, Limitations, and Societal Impact of Large Language Models
OpenAI summarizes a multidisciplinary workshop exploring the technical boundaries and societal risks associated with GPT-3 and other large language models.
Hugging Face Transformers v4.2.0 TensorFlow Performance and Serving Updates
Hugging Face Transformers v4.2.0 introduces significant computational performance gains for TensorFlow models and streamlined deployment via TensorFlow Serving using the SavedModel format.
OpenAI Scaling Kubernetes to 7,500 Nodes
OpenAI scaled a single Kubernetes cluster to 7,500 nodes to provide a simple, scalable infrastructure for machine learning research, overcoming challenges in networking, API server load, and monitoring.
Hugging Face Transformers ZeRO Integration via DeepSpeed and FairScale
Hugging Face Transformers v4.2.0 introduces experimental support for DeepSpeed and FairScale's ZeRO optimizations, enabling the training of larger models with higher batch sizes and reduced GPU memory requirements.
Hugging Face Accelerated Inference API Optimization
Hugging Face achieved a 100x speedup in transformer inference for its Accelerated Inference API by combining high-level library optimizations, Rust-based tokenization, and hardware-specific compilation.
CLIP: Connecting Text and Images
OpenAI introduced CLIP, a neural network that learns visual concepts from natural language supervision to enable zero-shot image classification across diverse datasets.
DALL·E: Creating images from text
OpenAI introduced DALL·E, a 12-billion parameter transformer model capable of generating diverse images from text descriptions by treating images and text as a single stream of tokens.
OpenAI Organizational Update December 2020
OpenAI announced the departure of VP of Research Dario Amodei and the appointment of Mira Murati as SVP of Research, Product, and Partnerships to better integrate safety and product development.
Leveraging Pre-trained Language Model Checkpoints for Encoder-Decoder Models – Hugging Face Blog Summary
Hugging Face’s blog post explains how to warm-start encoder-decoder models using pre‑trained BERT, RoBERTa, or GPT2 checkpoints, showing that this approach matches the performance of large pre‑trained seq2seq models while cutting training cost.
Porting fairseq WMT19 translation system to 🤗 Transformers
Hugging Face ported the fairseq WMT19 translation models (en‑ru, ru‑en, de‑en, en‑de) to the 🤗 Transformers library, allowing users to load and run these high‑quality translators with the standard Transformers API.
Hugging Face Transformers and Ray Tune Integration
Hugging Face Transformers 3.1 introduces an integration with Ray Tune, enabling users to easily implement advanced hyperparameter tuning algorithms like Population-Based Training and Bayesian Optimization.
Transformer-based Encoder-Decoder Models Hugging Face Blog Post 2020
Hugging Face’s 2020 blog post explains the transformer-based encoder-decoder architecture, detailing how it maps input sequences to variable-length outputs via encoder and decoder stacks, self-attention, cross-attention, and autoregressive generation, and shows how to use it with the 🤗Transformers library.
OpenAI Licenses GPT-3 Technology to Microsoft
OpenAI has licensed its GPT-3 language model technology to Microsoft for integration into Microsoft products and services, while maintaining independent API access for third-party developers.
Hugging Face pytorch_block_sparse Release
Hugging Face has released pytorch_block_sparse, a library providing BlockSparseLinear modules to create smaller and faster language models by reducing memory consumption and improving computation efficiency over standard PyTorch sparse matrices.
Generative Language Modeling for Automated Theorem Proving
OpenAI introduces GPT-f, a transformer-based language model designed for the Metamath formalization language that successfully contributed new proofs to the main Metamath library.
Learning to Summarize with Human Feedback
OpenAI demonstrates that reinforcement learning from human feedback (RLHF) allows smaller language models to outperform significantly larger models trained only via supervised learning in text summarization.
OpenAI Scholars 2020 Final Projects
OpenAI announced the final projects of the 2020 Scholars program, showcasing research in neural network interpretability, reinforcement learning, semantic parsing, and medical AI.