✷ The archive · 11 labs · 1,214 dispatches
The labs
No more opening a dozen official blogs every morning. First-hand releases from OpenAI, Anthropic, DeepMind and the rest, each with its substance pulled out.
DALL·E Outpainting Feature Release
OpenAI has introduced Outpainting for DALL·E, allowing users to extend images beyond their original borders using natural language descriptions while maintaining visual consistency.
OpenAI Announces Alignment Strategy Using RL from Human Feedback and InstructGPT
OpenAI detailed its alignment approach, highlighting RL from human feedback and InstructGPT as effective, low‑cost methods for making language models follow human intent while acknowledging current limitations and future goals for AI‑driven alignment research.
OpenAI Moderation Endpoint Release
OpenAI has released a free Moderation endpoint providing GPT-based classifiers to help developers detect and block prohibited content in their applications.
OpenAI Announces Efficient Training of Language Models for Fill-in-the-Middle
OpenAI released a new training method that enables autoregressive language models to perform fill‑in‑the‑middle tasks without sacrificing left‑to‑right generation quality, offering a simple and efficient way to add infilling capability.
A Hazard Analysis Framework for Code Synthesis Large Language Models
OpenAI has introduced a hazard analysis framework to identify and mitigate technical, social, political, and economic risks associated with the deployment of advanced code generation models like Codex.
OpenAI DALL·E beta release
OpenAI announced that DALL·E is now available in beta for 1 million invited users, offering free monthly credits and commercial usage rights.
Reducing Bias and Improving Safety in DALL·E 2
OpenAI has implemented a system-level technique for DALL·E 2 to increase the diversity of generated people and enhanced safety filters to prevent the creation of deceptive content.
DALL·E 2: Extending Creativity
OpenAI's DALL·E 2 research preview has been adopted by over 3,000 artists across 118 countries to integrate AI-generated imagery into diverse creative workflows ranging from medical support to high-end culinary arts.
DALL·E 2 Pre-training Mitigations
OpenAI implemented active learning for data filtering, a reweighting scheme to prevent bias amplification, and a clustering-based deduplication process to eliminate image regurgitation in DALL·E 2.
OpenAI Video PreTraining (VPT) for Minecraft
OpenAI introduced Video PreTraining (VPT), a semi-supervised imitation learning method that enables agents to learn complex behaviors in Minecraft by training on massive unlabeled video datasets of human play.
OpenAI Evolution through Large Models (ELM) Research
OpenAI introduces Evolution through Large Models (ELM), a method that uses LLMs to improve genetic programming by approximating human-like code mutations to generate functional programs in novel domains.
AI-written critiques help humans notice flaws
OpenAI research demonstrates that AI-generated critiques help human evaluators identify significantly more flaws in summaries, suggesting a scalable path for human supervision of complex AI tasks.
OpenAI Techniques for Training Large Neural Networks
OpenAI outlines the primary parallelism strategies—data, pipeline, tensor, and expert parallelism—and memory-saving techniques used to orchestrate GPU clusters for training massive AI models.
OpenAI, Cohere, and AI21 Labs Best Practices for Deploying Language Models
OpenAI, Cohere, and AI21 Labs have released a joint set of preliminary best practices for the responsible development and deployment of large language models to mitigate risks and prevent misuse.
OpenAI Research: Teaching Models to Express Their Uncertainty in Words
OpenAI researchers demonstrated that GPT-3 can be trained to express calibrated uncertainty about its answers in natural language without relying on model logits.
OpenAI Codex Applications and Integration
OpenAI Codex is a natural language-to-code system based on GPT-3 that powers over 70 applications, including GitHub Copilot, to increase developer productivity by converting English instructions into code.
OpenAI DALL·E 2 Research Preview Update
OpenAI is expanding access to DALL·E 2 to waitlisted users at a rate of 1,000 people per week following a limited research preview that generated over 3 million images.
OpenAI Leadership Team Update
OpenAI announced executive role changes for Greg Brockman, Brad Lightcap, Mira Murati, and Chris Clark to align leadership with the company's growth and strategic milestones.
Measuring Goodhart's Law in AI Alignment
OpenAI explores how optimizing proxy objectives in AI models can lead to a decline in true objective performance, a phenomenon known as Goodhart's Law, and proposes best-of-n sampling as a method for analysis.
Hierarchical Text-Conditional Image Generation with CLIP Latents
OpenAI introduces a two-stage generative model that uses a prior to create CLIP image embeddings from text and a decoder to generate images from those embeddings, improving diversity and enabling zero-shot image manipulation.
GPT-3 and Codex Edit and Insert Capabilities
OpenAI has released new versions of GPT-3 and Codex that can edit existing text and insert content into the middle of a document, moving beyond simple text completion.
OpenAI Economic Impacts Research Initiative
OpenAI has launched a call for expressions of interest from researchers and companies to study the economic impacts of large language models like GPT-3, ChatGPT, and DALL-E 2.
OpenAI Research Agenda for Economic Impacts of Code Generation Models
OpenAI has proposed a research agenda to study how code generation models like Codex affect productivity, employment, and economic inequality to inform deployment, system design, and public policy.
OpenAI Lessons Learned on Language Model Safety and Misuse
OpenAI shares insights from deploying GPT-3 and Codex, highlighting that real-world misuse often differs from theoretical risks and that safety research frequently enhances commercial utility.
OpenAI Formal Math Olympiad Problem Solving
OpenAI demonstrated the ability of its models to solve formal math olympiad problems from competitions like AMC12, AIME, and IMO using formal proof assistants.
OpenAI InstructGPT: Aligning Language Models to Follow Instructions
OpenAI introduced InstructGPT, a series of models trained using reinforcement learning from human feedback (RLHF) to better follow user intentions, increase truthfulness, and reduce toxicity compared to GPT-3.
OpenAI Text and Code Embeddings Release
OpenAI has released a new set of embedding models based on GPT-3 descendants that provide numerical representations of text and code to improve semantic search, clustering, and code retrieval.
OpenAI Text and Code Embeddings by Contrastive Pre-training
OpenAI has developed a method for creating high-quality text and code embeddings using contrastive pre-training on unsupervised data at scale, achieving state-of-the-art results in linear-probe classification and semantic search.
WebGPT: Improving GPT-3 Factual Accuracy via Web Browsing
OpenAI has fine-tuned GPT-3 into WebGPT, a prototype that uses a text-based web browser to research and cite sources, reducing hallucinations in open-ended question answering.
Customizing GPT-3 for Your Application
OpenAI has released fine-tuning capabilities for GPT-3, allowing developers to train the model on proprietary data to increase reliability, reduce costs, and improve latency.
OpenAI Residency Program Announcement
OpenAI has launched the OpenAI Residency, a six-month paid program designed to transition researchers and engineers from other fields into full-time AI roles at the company.
OpenAI API Access: GPT-3 Waitlist Removed
OpenAI has removed the waitlist for the GPT-3 API, allowing developers in supported countries to sign up and begin experimenting immediately.
OpenAI Solving Math Word Problems and the GSM8K Dataset
OpenAI developed a system using trained verifiers to solve grade school math word problems with nearly twice the accuracy of fine-tuned GPT-3, achieving 55% accuracy compared to 60% for 9-12 year old children.
OpenAI Summarizing Books with Human Feedback (2021)
OpenAI introduced a technique that combines reinforcement learning from human feedback with recursive task decomposition to generate summaries of entire books, addressing the alignment challenge of overseeing AI on difficult-to-evaluate tasks.
TruthfulQA: Measuring how models mimic human falsehoods
OpenAI introduces TruthfulQA, a benchmark designed to measure whether language models mimic common human misconceptions and falsehoods across 38 categories.
Helen Toner joins OpenAI board of directors
OpenAI appointed Helen Toner, an expert in AI policy and global strategy, to its board of directors on September 8, 2021, to strengthen its commitment to safe and responsible AI deployment.
OpenAI Codex: A GPT-3 Descendant for Code Generation
OpenAI Codex is a GPT-3 descendant trained on both natural language and billions of lines of public source code, enabling the generation of working code from English commands.
OpenAI Triton 1.0 Release
OpenAI has released Triton 1.0, an open-source Python-like language and compiler that allows researchers to write highly efficient GPU kernels without deep CUDA expertise.
Evaluating Large Language Models Trained on Code: OpenAI Codex
OpenAI introduces Codex, a GPT model fine-tuned on GitHub code, which solves 28.8% of HumanEval problems on a single attempt and up to 70.2% with repeated sampling.
OpenAI Research: Improving Language Model Behavior via Curated Dataset Fine-Tuning
OpenAI researchers found that fine-tuning GPT-3 on a small, curated dataset of fewer than 100 examples can significantly improve model adherence to specific behavioral values without compromising downstream performance.
OpenAI Scholars 2021: Final Projects
OpenAI announced the final projects of the 2021 Scholars program, featuring research on scaling laws, reward modeling, and reinforcement learning from a diverse group of researchers.
Will Hurd Joins OpenAI Board of Directors
OpenAI has appointed former U.S. Congressman Will Hurd to its board of directors to integrate public policy expertise with technical AI development.
GPT-3 API Ecosystem and Application Growth
OpenAI reports that over 300 applications and tens of thousands of developers are using GPT-3 to generate an average of 4.5 billion words per day across diverse industries.
Multimodal Neurons in CLIP
OpenAI discovered neurons in CLIP that respond to the same concept whether presented literally, symbolically, or conceptually, mirroring multimodal neurons found in the human brain.
Understanding the Capabilities, Limitations, and Societal Impact of Large Language Models
OpenAI summarizes a multidisciplinary workshop exploring the technical boundaries and societal risks associated with GPT-3 and other large language models.
OpenAI Scaling Kubernetes to 7,500 Nodes
OpenAI scaled a single Kubernetes cluster to 7,500 nodes to provide a simple, scalable infrastructure for machine learning research, overcoming challenges in networking, API server load, and monitoring.
CLIP: Connecting Text and Images
OpenAI introduced CLIP, a neural network that learns visual concepts from natural language supervision to enable zero-shot image classification across diverse datasets.
DALL·E: Creating images from text
OpenAI introduced DALL·E, a 12-billion parameter transformer model capable of generating diverse images from text descriptions by treating images and text as a single stream of tokens.
OpenAI Organizational Update December 2020
OpenAI announced the departure of VP of Research Dario Amodei and the appointment of Mira Murati as SVP of Research, Product, and Partnerships to better integrate safety and product development.
OpenAI Licenses GPT-3 Technology to Microsoft
OpenAI has licensed its GPT-3 language model technology to Microsoft for integration into Microsoft products and services, while maintaining independent API access for third-party developers.