✷ The archive · 11 labs · 3,433 dispatches
The labs
No more opening a dozen official blogs every morning. First-hand releases from OpenAI, Anthropic, DeepMind and the rest, each with its substance pulled out.
Perceiver IO: A Scalable, Fully-Attentional Model for Any Modality
Perceiver IO is a Transformer-based architecture that decouples compute from input size by using a latent space, enabling it to process text, images, audio, video, and point clouds without quadratic scaling issues.
Customizing GPT-3 for Your Application
OpenAI has released fine-tuning capabilities for GPT-3, allowing developers to train the model on proprietary data to increase reliability, reduce costs, and improve latency.
Training CodeParrot from Scratch
Hugging Face introduces CodeParrot, a GPT-2 based model trained from scratch on a cleaned dataset of 20 million Python files to enable Python code auto-completion.
Hugging Face Snowball Fight ML-Agents Environment
Hugging Face has released Snowball Fight 1vs1, its first custom Deep Reinforcement Learning environment built with Unity ML-Agents and hosted on Hugging Face Spaces.
Anthropic Research: A General Language Assistant as a Laboratory for Alignment
Anthropic explores methods to create a helpful, honest, and harmless general-purpose language assistant, finding that ranked preference modeling scales more effectively than imitation learning or binary discrimination.
OpenAI Residency Program Announcement
OpenAI has launched the OpenAI Residency, a six-month paid program designed to transition researchers and engineers from other fields into full-time AI roles at the company.
Hugging Face Optimum for Graphcore IPU Integration
Hugging Face has integrated the Optimum library with Graphcore Intelligence Processing Units (IPUs) to accelerate Transformer models, starting with an optimized BERT implementation.
Hugging Face Data Measurements Tool
Hugging Face has released the Data Measurements Tool, an open-source Python library and no-code interface designed to help developers analyze, curate, and compare ML datasets for more responsible AI development.
Accelerating PyTorch Distributed Fine-Tuning with Intel Technologies
Hugging Face demonstrates how to accelerate PyTorch training by distributing fine-tuning jobs across a cluster of Intel Xeon Scalable CPU servers using the Intel extension for PyTorch and oneCCL.
OpenAI API Access: GPT-3 Waitlist Removed
OpenAI has removed the waitlist for the GPT-3 API, allowing developers in supported countries to sign up and begin experimenting immediately.
Fine-Tuning XLS-R for Low-Resource Automatic Speech Recognition
Hugging Face provides a technical guide on fine-tuning XLS-R, a cross-lingual speech representation model, for low-resource Automatic Speech Recognition (ASR) using the Transformers library.
Scaling up BERT-like model Inference on modern CPU - Part 2
Hugging Face explores software-level optimizations for BERT-like models on Intel Ice Lake Xeon CPUs, demonstrating how memory allocators, parallelization libraries, and Bayesian optimization can significantly reduce inference latency.
OpenAI Solving Math Word Problems and the GSM8K Dataset
OpenAI developed a system using trained verifiers to solve grade school math word problems with nearly twice the accuracy of fine-tuned GPT-3, achieving 55% accuracy compared to 60% for 9-12 year old children.
Hugging Face Course Part 2 and Community Event Launch
Hugging Face announced the release of Part 2 of the Hugging Face Course on November 15, 2021, accompanied by a community event featuring technical talks and hands-on projects.
Large Language Models: A New Moore's Law?
Hugging Face critiques the trend of exponentially increasing model sizes, such as the 530B parameter Megatron-Turing NLG, and advocates for pragmatic, efficient alternatives like distillation and fine-tuning.
Hugging Face Sentence Embedding Models with 1B Training Pairs
Hugging Face developed state-of-the-art general-purpose sentence embedding models by training on up to 1 billion sentence pairs using JAX/Flax and TPU infrastructure.
Hugging Face: The Age of Machine Learning As Code
Hugging Face advocates for treating machine learning as a software engineering discipline by adopting MLOps, leveraging the general-purpose Transformer architecture, and prioritizing production deployment over sandbox experiments.
Fine-tuning CLIP for Remote Sensing and Satellite Imagery
A team of researchers fine-tuned OpenAI's CLIP model using the RSICD dataset and other satellite imagery to significantly improve text-to-image retrieval for remote sensing applications.
Hugging Face Spaces and Gradio Integration
Hugging Face has integrated Gradio into Spaces, allowing users to easily host and showcase machine learning model demos using the Inference API or custom model checkpoints.
Hosting Models and Datasets on Hugging Face Spaces using Streamlit
Hugging Face Spaces integrates with Streamlit to allow users to quickly build and host interactive demos for machine learning models and data visualizations.
Hugging Face Summer 2021 Update
Hugging Face announced a series of Hub enhancements, including the Spaces Beta for ML demo hosting, TensorBoard integration, and the launch of the Optimum library for hardware acceleration.
OpenAI Summarizing Books with Human Feedback (2021)
OpenAI introduced a technique that combines reinforcement learning from human feedback with recursive task decomposition to generate summaries of entire books, addressing the alignment challenge of overseeing AI on difficult-to-evaluate tasks.
Hugging Face Optimum Release
Hugging Face has introduced Optimum, an open-source toolkit designed to optimize Transformer models for production performance across various hardware platforms.
Hugging Face and Graphcore Partnership for IPU-Optimized Transformers
Hugging Face and Graphcore have partnered to integrate Intelligence Processing Units (IPUs) into the Hugging Face ecosystem via the Hardware Partner Program and the Optimum library to accelerate Transformer model deployment.
TruthfulQA: Measuring how models mimic human falsehoods
OpenAI introduces TruthfulQA, a benchmark designed to measure whether language models mimic common human misconceptions and falsehoods across 38 categories.
Helen Toner joins OpenAI board of directors
OpenAI appointed Helen Toner, an expert in AI policy and global strategy, to its board of directors on September 8, 2021, to strengthen its commitment to safe and responsible AI deployment.
OpenAI Codex: A GPT-3 Descendant for Code Generation
OpenAI Codex is a GPT-3 descendant trained on both natural language and billions of lines of public source code, enabling the generation of working code from English commands.
OpenAI Triton 1.0 Release
OpenAI has released Triton 1.0, an open-source Python-like language and compiler that allows researchers to write highly efficient GPU kernels without deep CUDA expertise.
Hugging Face DeDLOC: Collaborative Training of Language Models over the Internet
Hugging Face introduces DeDLOC, a distributed training method that enables volunteers to collaboratively pretrain large language models over the internet by adapting to varying network and hardware constraints.
spaCy Integration with Hugging Face Hub
Hugging Face has integrated spaCy into the Hugging Face Hub, allowing users to share, discover, and deploy spaCy pipelines via a unified platform.
Deploy Hugging Face models easily with Amazon SageMaker
Hugging Face and Amazon SageMaker have introduced new Inference Deep Learning Containers (DLCs) and an Inference Toolkit to simplify the deployment of Transformer models to production-ready endpoints.
Evaluating Large Language Models Trained on Code: OpenAI Codex
OpenAI introduces Codex, a GPT model fine-tuned on GitHub code, which solves 28.8% of HumanEval problems on a single attempt and up to 70.2% with repeated sampling.
Sentence Transformers Integration in the Hugging Face Hub
Hugging Face has integrated Sentence Transformers into the Hub, providing over 90 pretrained models for 100+ languages and new interactive widgets for feature extraction and sentence similarity.
OpenAI Research: Improving Language Model Behavior via Curated Dataset Fine-Tuning
OpenAI researchers found that fine-tuning GPT-3 on a small, curated dataset of fewer than 100 examples can significantly improve model adherence to specific behavioral values without compromising downstream performance.
Few-Shot Learning with GPT-Neo and Hugging Face Accelerated Inference API
Hugging Face explores the application of few-shot learning using the open-source GPT-Neo model and the Accelerated Inference API to enable task generalization without extensive labeled data.
Anthropic Series A Funding for Reliable General AI Systems
Anthropic has raised $124 million in Series A funding to develop large-scale AI systems that are steerable, interpretable, and robust.
Using & Mixing Hugging Face Models with Gradio 2.0
Gradio 2.0 enables machine learning developers to load and deploy Hugging Face models as GUIs with a single line of code, supporting both parallel and serial model composition.
OpenAI Scholars 2021: Final Projects
OpenAI announced the final projects of the 2021 Scholars program, featuring research on scaling laws, reward modeling, and reinforcement learning from a diverse group of researchers.
Will Hurd Joins OpenAI Board of Directors
OpenAI has appointed former U.S. Congressman Will Hurd to its board of directors to integrate public policy expertise with technical AI development.
Scaling-up BERT Inference on CPU (Part 1)
Hugging Face explores hardware-level optimizations for BERT inference on modern CPUs, demonstrating that throughput can be scaled linearly by using multiple independent model instances bound to specific physical cores via NUMA-aware affinity.
Hugging Face Accelerate Library Release
Hugging Face has released Accelerate, a PyTorch library that allows users to run raw training scripts on any device configuration, including multi-GPU and TPU, without rewriting boilerplate code.
Distributed Training of BART and T5 for Summarization via Hugging Face and Amazon SageMaker
Hugging Face and Amazon SageMaker have integrated to provide optimized Deep Learning Containers and a dedicated HuggingFace estimator to simplify distributed training of Transformers models like BART and T5.
Understanding BigBird's Block Sparse Attention
BigBird introduces block sparse attention to reduce the computational complexity of Transformers from quadratic to linear, enabling the processing of sequences up to 4096 tokens.
GPT-3 API Ecosystem and Application Growth
OpenAI reports that over 300 applications and tens of thousands of developers are using GPT-3 to generate an average of 4.5 billion words per day across diverse industries.
Amazon SageMaker and Hugging Face Partnership
Hugging Face and Amazon have partnered to integrate Hugging Face Transformers into Amazon SageMaker via dedicated Deep Learning Containers (DLCs) and a Python SDK extension to accelerate NLP model training and deployment.
Deploying a Hugging Face Transformers Sentiment Analysis Pipeline on Google Cloud Run
A community member demonstrated how to serve a Hugging Face sentiment‑analysis pipeline on Google Cloud Run using a DistilBERT model, achieving sub‑5‑second latency with minimal monthly cost.
Fine-Tuning Wav2Vec2 for English ASR with Hugging Face Transformers
Hugging Face provides a detailed guide on fine-tuning the Wav2Vec2 pretrained speech model for English Automatic Speech Recognition (ASR) using Connectionist Temporal Classification (CTC) loss.
Hugging Face Reads: Long-range Transformers
Hugging Face analyzes four key architectures—Longformer, Compressive Transformer, Linformer, and Performer—designed to reduce the quadratic memory and time complexity of standard Transformer self-attention to linear complexity.
Multimodal Neurons in CLIP
OpenAI discovered neurons in CLIP that respond to the same concept whether presented literally, symbolically, or conceptually, mirroring multimodal neurons found in the human brain.
Hugging Face: Simple Considerations for Building Neural Networks
Hugging Face provides a framework for building and debugging neural networks by prioritizing data analysis, simple baselines, and rigorous implementation checks over blind hyperparameter tuning.