✷ The archive · 11 labs · 3,062 dispatches
The labs
No more opening a dozen official blogs every morning. First-hand releases from OpenAI, Anthropic, DeepMind and the rest, each with its substance pulled out.
Machine Learning Experts: An Interview with Margaret Mitchell
Dr. Margaret Mitchell discusses the critical role of ethical AI, the development of Model Cards for transparency, and the necessity of diversity and inclusion in machine learning development.
Hugging Face AI Research Residency Program Announcement
Hugging Face has launched a 9-month AI Research Residency Program designed to help aspiring researchers develop machine learning techniques and publish open-source work alongside the Hugging Face Science Team.
Fine-Tuning SegFormer for Semantic Segmentation with Custom Datasets
Hugging Face provides a technical guide on fine-tuning the SegFormer model using the Transformers library to perform pixel-level image classification for custom use cases like autonomous sidewalk navigation.
Image search with Hugging Face datasets
Hugging Face demonstrates how to build an image search application by combining the datasets library, CLIP embeddings, and FAISS for efficient similarity search.
Accelerate BERT Inference with Hugging Face Transformers and AWS Inferentia
Hugging Face provides a guide on using AWS Inferentia and the Neuron SDK to accelerate BERT inference, achieving 5-6ms latency on Amazon SageMaker.
GPT-3 and Codex Edit and Insert Capabilities
OpenAI has released new versions of GPT-3 and Codex that can edit existing text and insert content into the middle of a document, moving beyond simple text completion.
Guiding Text Generation with Constrained Beam Search in Hugging Face Transformers
Hugging Face introduced constrained beam search to the Transformers library, allowing users to force specific words or phrases into generated text while maintaining linguistic coherence.
Anthropic In-context Learning and Induction Heads
Anthropic announced a research post titled “In-context Learning and Induction Heads” on March 8 2022, but the page contains only related links and no substantive technical details.
OpenAI Economic Impacts Research Initiative
OpenAI has launched a call for expressions of interest from researchers and companies to study the economic impacts of large language models like GPT-3, ChatGPT, and DALL-E 2.
OpenAI Research Agenda for Economic Impacts of Code Generation Models
OpenAI has proposed a research agenda to study how code generation models like Codex affect productivity, employment, and economic inequality to inform deployment, system design, and public policy.
OpenAI Lessons Learned on Language Model Safety and Misuse
OpenAI shares insights from deploying GPT-3 and Codex, highlighting that real-world misuse often differs from theoretical risks and that safety research frequently enhances commercial utility.
BERT 101: Understanding the Bidirectional Encoder Representations from Transformers
BERT is a bidirectional Transformer-based model developed by Google AI Language that achieves state-of-the-art performance across 11+ common NLP tasks by leveraging unsupervised pre-training on massive datasets.
Anthropic Research: Predictability and Surprise in Large Generative Models
Anthropic identifies a tension between the predictable scaling of loss in large generative models and the unpredictable emergence of specific capabilities and outputs, creating challenges for AI safety and policy.
Fine-Tuning Vision Transformer (ViT) for Image Classification with Hugging Face Transformers
Hugging Face provides a guide on fine-tuning the Vision Transformer (ViT) model for image classification using the datasets and transformers libraries, demonstrating the process with the beans dataset.
OpenAI Formal Math Olympiad Problem Solving
OpenAI demonstrated the ability of its models to solve formal math olympiad problems from competitions like AMC12, AIME, and IMO using formal proof assistants.
Getting Started with Sentiment Analysis using Python
Hugging Face provides a comprehensive guide on implementing sentiment analysis using Python, leveraging pre-trained models from the Hugging Face Hub, the Trainer API for fine-tuning, and the no-code AutoNLP tool.
Automatic Speech Recognition for Large Files with Wav2Vec2 in Transformers
Hugging Face implements a striding technique leveraging the Connectionist Temporal Classification (CTC) architecture of Wav2Vec2 to enable high-quality ASR on arbitrarily long audio files and live inference.
OpenAI InstructGPT: Aligning Language Models to Follow Instructions
OpenAI introduced InstructGPT, a series of models trained using reinforcement learning from human feedback (RLHF) to better follow user intentions, increase truthfulness, and reduce toxicity compared to GPT-3.
OpenAI Text and Code Embeddings Release
OpenAI has released a new set of embedding models based on GPT-3 descendants that provide numerical representations of text and code to improve semantic search, clustering, and code retrieval.
Hugging Face Hub Search API Updates
Hugging Face has introduced new programmatic search features to the huggingface_hub library, including ModelSearchArguments and ModelFilter, to simplify how users find models and datasets without leaving their IDE.
OpenAI Text and Code Embeddings by Contrastive Pre-training
OpenAI has developed a method for creating high-quality text and code embeddings using contrastive pre-training on unsupervised data at scale, achieving state-of-the-art results in linear-probe classification and semantic search.
Stable-Baselines3 Integration with Hugging Face Hub
Hugging Face has integrated Stable-Baselines3, allowing users to host, share, and load PyTorch Deep Reinforcement Learning models directly from the Hugging Face Hub.
Hugging Face Infinity CPU Performance Case Study
Hugging Face Infinity achieves up to 800% higher throughput and millisecond latency on Intel Ice Lake Xeon CPUs compared to vanilla Transformers, enabling cost-effective, real-time Transformer deployments on CPU infrastructure.
Boosting Wav2Vec2 with n-grams in Hugging Face Transformers
Hugging Face has integrated the pyctcdecode library into the Transformers library, enabling Wav2Vec2 models to be boosted with n-gram language models to significantly reduce spelling errors and Word Error Rate (WER).
Deploying GPT-J 6B on Amazon SageMaker with Hugging Face Transformers
Hugging Face provides a method to deploy EleutherAI's GPT-J 6B model on Amazon SageMaker, reducing model load times from over three minutes to under eight seconds using torch.save.
Active Learning with AutoNLP and Prodigy
Hugging Face demonstrates how to build an active learning pipeline for Named Entity Recognition (NER) by combining the AutoNLP automated training framework with the Prodigy annotation tool.
Anthropic Announces Mathematical Framework for Transformer Circuits
Anthropic released a brief announcement of a new mathematical framework for transformer circuits, highlighting its potential to deepen understanding of model behavior but providing no technical details in the post.
Hugging Face Acquires Gradio
Hugging Face has acquired Gradio to integrate easy-to-build machine learning demos and GUIs into its ecosystem, expanding ML accessibility to non-technical users.
WebGPT: Improving GPT-3 Factual Accuracy via Web Browsing
OpenAI has fine-tuned GPT-3 into WebGPT, a prototype that uses a text-based web browser to research and cite sources, reducing hallucinations in open-ended question answering.
Perceiver IO: A Scalable, Fully-Attentional Model for Any Modality
Perceiver IO is a Transformer-based architecture that decouples compute from input size by using a latent space, enabling it to process text, images, audio, video, and point clouds without quadratic scaling issues.
Customizing GPT-3 for Your Application
OpenAI has released fine-tuning capabilities for GPT-3, allowing developers to train the model on proprietary data to increase reliability, reduce costs, and improve latency.
Training CodeParrot from Scratch
Hugging Face introduces CodeParrot, a GPT-2 based model trained from scratch on a cleaned dataset of 20 million Python files to enable Python code auto-completion.
Hugging Face Snowball Fight ML-Agents Environment
Hugging Face has released Snowball Fight 1vs1, its first custom Deep Reinforcement Learning environment built with Unity ML-Agents and hosted on Hugging Face Spaces.
Anthropic Research: A General Language Assistant as a Laboratory for Alignment
Anthropic explores methods to create a helpful, honest, and harmless general-purpose language assistant, finding that ranked preference modeling scales more effectively than imitation learning or binary discrimination.
OpenAI Residency Program Announcement
OpenAI has launched the OpenAI Residency, a six-month paid program designed to transition researchers and engineers from other fields into full-time AI roles at the company.
Hugging Face Optimum for Graphcore IPU Integration
Hugging Face has integrated the Optimum library with Graphcore Intelligence Processing Units (IPUs) to accelerate Transformer models, starting with an optimized BERT implementation.
Hugging Face Data Measurements Tool
Hugging Face has released the Data Measurements Tool, an open-source Python library and no-code interface designed to help developers analyze, curate, and compare ML datasets for more responsible AI development.
Accelerating PyTorch Distributed Fine-Tuning with Intel Technologies
Hugging Face demonstrates how to accelerate PyTorch training by distributing fine-tuning jobs across a cluster of Intel Xeon Scalable CPU servers using the Intel extension for PyTorch and oneCCL.
OpenAI API Access: GPT-3 Waitlist Removed
OpenAI has removed the waitlist for the GPT-3 API, allowing developers in supported countries to sign up and begin experimenting immediately.
Fine-Tuning XLS-R for Low-Resource Automatic Speech Recognition
Hugging Face provides a technical guide on fine-tuning XLS-R, a cross-lingual speech representation model, for low-resource Automatic Speech Recognition (ASR) using the Transformers library.
Scaling up BERT-like model Inference on modern CPU - Part 2
Hugging Face explores software-level optimizations for BERT-like models on Intel Ice Lake Xeon CPUs, demonstrating how memory allocators, parallelization libraries, and Bayesian optimization can significantly reduce inference latency.
OpenAI Solving Math Word Problems and the GSM8K Dataset
OpenAI developed a system using trained verifiers to solve grade school math word problems with nearly twice the accuracy of fine-tuned GPT-3, achieving 55% accuracy compared to 60% for 9-12 year old children.
Hugging Face Course Part 2 and Community Event Launch
Hugging Face announced the release of Part 2 of the Hugging Face Course on November 15, 2021, accompanied by a community event featuring technical talks and hands-on projects.
Large Language Models: A New Moore's Law?
Hugging Face critiques the trend of exponentially increasing model sizes, such as the 530B parameter Megatron-Turing NLG, and advocates for pragmatic, efficient alternatives like distillation and fine-tuning.
Hugging Face Sentence Embedding Models with 1B Training Pairs
Hugging Face developed state-of-the-art general-purpose sentence embedding models by training on up to 1 billion sentence pairs using JAX/Flax and TPU infrastructure.
Hugging Face: The Age of Machine Learning As Code
Hugging Face advocates for treating machine learning as a software engineering discipline by adopting MLOps, leveraging the general-purpose Transformer architecture, and prioritizing production deployment over sandbox experiments.
Fine-tuning CLIP for Remote Sensing and Satellite Imagery
A team of researchers fine-tuned OpenAI's CLIP model using the RSICD dataset and other satellite imagery to significantly improve text-to-image retrieval for remote sensing applications.
Hugging Face Spaces and Gradio Integration
Hugging Face has integrated Gradio into Spaces, allowing users to easily host and showcase machine learning model demos using the Inference API or custom model checkpoints.
Hosting Models and Datasets on Hugging Face Spaces using Streamlit
Hugging Face Spaces integrates with Streamlit to allow users to quickly build and host interactive demos for machine learning models and data visualizations.
Hugging Face Summer 2021 Update
Hugging Face announced a series of Hub enhancements, including the Spaces Beta for ML demo hosting, TensorBoard integration, and the launch of the Optimum library for hardware acceleration.