OpenAI Text and Code Embeddings Release
OpenAI has released a new set of embedding models based on GPT-3 descendants that provide numerical representations of text and code to improve semantic search, clustering, and code retrieval.
Hugging Face Hub Search API Updates
Hugging Face has introduced new programmatic search features to the huggingface_hub library, including ModelSearchArguments and ModelFilter, to simplify how users find models and datasets without leaving their IDE.
OpenAI Text and Code Embeddings by Contrastive Pre-training
OpenAI has developed a method for creating high-quality text and code embeddings using contrastive pre-training on unsupervised data at scale, achieving state-of-the-art results in linear-probe classification and semantic search.
Stable-Baselines3 Integration with Hugging Face Hub
Hugging Face has integrated Stable-Baselines3, allowing users to host, share, and load PyTorch Deep Reinforcement Learning models directly from the Hugging Face Hub.
Hugging Face Infinity CPU Performance Case Study
Hugging Face Infinity achieves up to 800% higher throughput and millisecond latency on Intel Ice Lake Xeon CPUs compared to vanilla Transformers, enabling cost-effective, real-time Transformer deployments on CPU infrastructure.
Boosting Wav2Vec2 with n-grams in Hugging Face Transformers
Hugging Face has integrated the pyctcdecode library into the Transformers library, enabling Wav2Vec2 models to be boosted with n-gram language models to significantly reduce spelling errors and Word Error Rate (WER).
Deploying GPT-J 6B on Amazon SageMaker with Hugging Face Transformers
Hugging Face provides a method to deploy EleutherAI's GPT-J 6B model on Amazon SageMaker, reducing model load times from over three minutes to under eight seconds using torch.save.
Active Learning with AutoNLP and Prodigy
Hugging Face demonstrates how to build an active learning pipeline for Named Entity Recognition (NER) by combining the AutoNLP automated training framework with the Prodigy annotation tool.
Hugging Face Acquires Gradio
Hugging Face has acquired Gradio to integrate easy-to-build machine learning demos and GUIs into its ecosystem, expanding ML accessibility to non-technical users.
WebGPT: Improving GPT-3 Factual Accuracy via Web Browsing
OpenAI has fine-tuned GPT-3 into WebGPT, a prototype that uses a text-based web browser to research and cite sources, reducing hallucinations in open-ended question answering.
Perceiver IO: A Scalable, Fully-Attentional Model for Any Modality
Perceiver IO is a Transformer-based architecture that decouples compute from input size by using a latent space, enabling it to process text, images, audio, video, and point clouds without quadratic scaling issues.
Customizing GPT-3 for Your Application
OpenAI has released fine-tuning capabilities for GPT-3, allowing developers to train the model on proprietary data to increase reliability, reduce costs, and improve latency.
Training CodeParrot from Scratch
Hugging Face introduces CodeParrot, a GPT-2 based model trained from scratch on a cleaned dataset of 20 million Python files to enable Python code auto-completion.
Hugging Face Snowball Fight ML-Agents Environment
Hugging Face has released Snowball Fight 1vs1, its first custom Deep Reinforcement Learning environment built with Unity ML-Agents and hosted on Hugging Face Spaces.
OpenAI Residency Program Announcement
OpenAI has launched the OpenAI Residency, a six-month paid program designed to transition researchers and engineers from other fields into full-time AI roles at the company.
Hugging Face Optimum for Graphcore IPU Integration
Hugging Face has integrated the Optimum library with Graphcore Intelligence Processing Units (IPUs) to accelerate Transformer models, starting with an optimized BERT implementation.
Hugging Face Data Measurements Tool
Hugging Face has released the Data Measurements Tool, an open-source Python library and no-code interface designed to help developers analyze, curate, and compare ML datasets for more responsible AI development.
Accelerating PyTorch Distributed Fine-Tuning with Intel Technologies
Hugging Face demonstrates how to accelerate PyTorch training by distributing fine-tuning jobs across a cluster of Intel Xeon Scalable CPU servers using the Intel extension for PyTorch and oneCCL.
OpenAI API Access: GPT-3 Waitlist Removed
OpenAI has removed the waitlist for the GPT-3 API, allowing developers in supported countries to sign up and begin experimenting immediately.
Fine-Tuning XLS-R for Low-Resource Automatic Speech Recognition
Hugging Face provides a technical guide on fine-tuning XLS-R, a cross-lingual speech representation model, for low-resource Automatic Speech Recognition (ASR) using the Transformers library.
Scaling up BERT-like model Inference on modern CPU - Part 2
Hugging Face explores software-level optimizations for BERT-like models on Intel Ice Lake Xeon CPUs, demonstrating how memory allocators, parallelization libraries, and Bayesian optimization can significantly reduce inference latency.
OpenAI Solving Math Word Problems and the GSM8K Dataset
OpenAI developed a system using trained verifiers to solve grade school math word problems with nearly twice the accuracy of fine-tuned GPT-3, achieving 55% accuracy compared to 60% for 9-12 year old children.
Hugging Face Course Part 2 and Community Event Launch
Hugging Face announced the release of Part 2 of the Hugging Face Course on November 15, 2021, accompanied by a community event featuring technical talks and hands-on projects.
Large Language Models: A New Moore's Law?
Hugging Face critiques the trend of exponentially increasing model sizes, such as the 530B parameter Megatron-Turing NLG, and advocates for pragmatic, efficient alternatives like distillation and fine-tuning.
Hugging Face Sentence Embedding Models with 1B Training Pairs
Hugging Face developed state-of-the-art general-purpose sentence embedding models by training on up to 1 billion sentence pairs using JAX/Flax and TPU infrastructure.
Hugging Face: The Age of Machine Learning As Code
Hugging Face advocates for treating machine learning as a software engineering discipline by adopting MLOps, leveraging the general-purpose Transformer architecture, and prioritizing production deployment over sandbox experiments.
Fine-tuning CLIP for Remote Sensing and Satellite Imagery
A team of researchers fine-tuned OpenAI's CLIP model using the RSICD dataset and other satellite imagery to significantly improve text-to-image retrieval for remote sensing applications.
Hugging Face Spaces and Gradio Integration
Hugging Face has integrated Gradio into Spaces, allowing users to easily host and showcase machine learning model demos using the Inference API or custom model checkpoints.
Hosting Models and Datasets on Hugging Face Spaces using Streamlit
Hugging Face Spaces integrates with Streamlit to allow users to quickly build and host interactive demos for machine learning models and data visualizations.
Hugging Face Summer 2021 Update
Hugging Face announced a series of Hub enhancements, including the Spaces Beta for ML demo hosting, TensorBoard integration, and the launch of the Optimum library for hardware acceleration.
OpenAI Summarizing Books with Human Feedback (2021)
OpenAI introduced a technique that combines reinforcement learning from human feedback with recursive task decomposition to generate summaries of entire books, addressing the alignment challenge of overseeing AI on difficult-to-evaluate tasks.
Hugging Face Optimum Release
Hugging Face has introduced Optimum, an open-source toolkit designed to optimize Transformer models for production performance across various hardware platforms.
Hugging Face and Graphcore Partnership for IPU-Optimized Transformers
Hugging Face and Graphcore have partnered to integrate Intelligence Processing Units (IPUs) into the Hugging Face ecosystem via the Hardware Partner Program and the Optimum library to accelerate Transformer model deployment.
TruthfulQA: Measuring how models mimic human falsehoods
OpenAI introduces TruthfulQA, a benchmark designed to measure whether language models mimic common human misconceptions and falsehoods across 38 categories.
Helen Toner joins OpenAI board of directors
OpenAI appointed Helen Toner, an expert in AI policy and global strategy, to its board of directors on September 8, 2021, to strengthen its commitment to safe and responsible AI deployment.
OpenAI Codex: A GPT-3 Descendant for Code Generation
OpenAI Codex is a GPT-3 descendant trained on both natural language and billions of lines of public source code, enabling the generation of working code from English commands.
OpenAI Triton 1.0 Release
OpenAI has released Triton 1.0, an open-source Python-like language and compiler that allows researchers to write highly efficient GPU kernels without deep CUDA expertise.
Hugging Face DeDLOC: Collaborative Training of Language Models over the Internet
Hugging Face introduces DeDLOC, a distributed training method that enables volunteers to collaboratively pretrain large language models over the internet by adapting to varying network and hardware constraints.
spaCy Integration with Hugging Face Hub
Hugging Face has integrated spaCy into the Hugging Face Hub, allowing users to share, discover, and deploy spaCy pipelines via a unified platform.
Deploy Hugging Face models easily with Amazon SageMaker
Hugging Face and Amazon SageMaker have introduced new Inference Deep Learning Containers (DLCs) and an Inference Toolkit to simplify the deployment of Transformer models to production-ready endpoints.
Evaluating Large Language Models Trained on Code: OpenAI Codex
OpenAI introduces Codex, a GPT model fine-tuned on GitHub code, which solves 28.8% of HumanEval problems on a single attempt and up to 70.2% with repeated sampling.
Sentence Transformers Integration in the Hugging Face Hub
Hugging Face has integrated Sentence Transformers into the Hub, providing over 90 pretrained models for 100+ languages and new interactive widgets for feature extraction and sentence similarity.
OpenAI Research: Improving Language Model Behavior via Curated Dataset Fine-Tuning
OpenAI researchers found that fine-tuning GPT-3 on a small, curated dataset of fewer than 100 examples can significantly improve model adherence to specific behavioral values without compromising downstream performance.
Few-Shot Learning with GPT-Neo and Hugging Face Accelerated Inference API
Hugging Face explores the application of few-shot learning using the open-source GPT-Neo model and the Accelerated Inference API to enable task generalization without extensive labeled data.
Using & Mixing Hugging Face Models with Gradio 2.0
Gradio 2.0 enables machine learning developers to load and deploy Hugging Face models as GUIs with a single line of code, supporting both parallel and serial model composition.
OpenAI Scholars 2021: Final Projects
OpenAI announced the final projects of the 2021 Scholars program, featuring research on scaling laws, reward modeling, and reinforcement learning from a diverse group of researchers.
Will Hurd Joins OpenAI Board of Directors
OpenAI has appointed former U.S. Congressman Will Hurd to its board of directors to integrate public policy expertise with technical AI development.
Scaling-up BERT Inference on CPU (Part 1)
Hugging Face explores hardware-level optimizations for BERT inference on modern CPUs, demonstrating that throughput can be scaled linearly by using multiple independent model instances bound to specific physical cores via NUMA-aware affinity.
Hugging Face Accelerate Library Release
Hugging Face has released Accelerate, a PyTorch library that allows users to run raw training scripts on any device configuration, including multi-GPU and TPU, without rewriting boilerplate code.
Distributed Training of BART and T5 for Summarization via Hugging Face and Amazon SageMaker
Hugging Face and Amazon SageMaker have integrated to provide optimized Deep Learning Containers and a dedicated HuggingFace estimator to simplify distributed training of Transformers models like BART and T5.