ChatGPT Plugins Release

OpenAI has implemented initial support for plugins in ChatGPT, enabling the language model to access up-to-date information, execute computations, and utilize third-party services. This expansion transforms ChatGPT from a text-generation engine into a system capable of interacting with external data and performing constrained actions on behalf of users.

Core Capabilities and Purpose

Plugins serve as "eyes and ears" for language models, overcoming the inherent limitations of static training data. By providing access to information that is too recent, personal, or specific for a training corpus, plugins allow ChatGPT to:

  • Access Real-Time Data: Retrieve current information from the internet or specific databases.
  • Perform Computations: Execute precise mathematical and technical calculations.
  • Execute Actions: Perform safe, constrained tasks using third-party service integrations.

OpenAI is deploying these tools iteratively, starting with a small set of users and ChatGPT Plus subscribers, to study real-world impact and safety challenges before expanding access to API users.

First-Party Plugins: Browsing and Code Interpreter

OpenAI has launched two primary internal plugins to enhance the model's foundational capabilities:

Web Browsing

The browsing plugin allows ChatGPT to read information from the internet, expanding its knowledge beyond the training corpus.

  • Technical Implementation: The plugin uses the Bing search API and is limited to making GET requests to reduce security risks. It operates within an isolated service separate from OpenAI's main infrastructure.
  • Web Citizenship: The plugin uses the user-agent token ChatGPT-User and honors robots.txt files to respect content creators.
  • Transparency: The system cites its sources and lists visited websites, allowing users to verify accuracy and provide credit to creators.

Code Interpreter

The code interpreter provides ChatGPT with a working Python interpreter in a sandboxed, firewalled execution environment with ephemeral disk space.

  • Functionality: It supports file uploads and downloads, allowing the model to solve quantitative and qualitative mathematical problems, perform data analysis and visualization, and convert files between formats.
  • Safety: To prevent unintended side effects, the environment is secured with strict network controls that disable external internet access from executed code.

Open-Source Retrieval Plugin

OpenAI has open-sourced a retrieval plugin that enables ChatGPT to access personal or organizational information sources with permission.

  • Mechanism: The plugin leverages OpenAI embeddings and allows developers to choose a vector database (such as Milvus, Pinecone, Qdrant, Redis, Weaviate, or Zilliz) for indexing and searching documents.
  • Security: Because the plugin is a self-hosted solution that only adds results into the ChatGPT session without external effects, the primary security risks are centered on data authorization and privacy.

Third-Party Plugin Ecosystem

Third-party plugins are designed to be invoked by the language model based on a manifest file containing a machine-readable description of capabilities and user-facing documentation.

Development Process

Developers create plugins by building an API (new, existing, or a wrapper) and providing an OpenAPI specification and a manifest file. When a user enables a plugin, its documentation is provided to the model as part of the conversation context, allowing the model to determine when and how to invoke the API.

Initial Partners

Early collaborators include Expedia, FiscalNote, Instacart, KAYAK, Klarna, Milo, OpenTable, Shopify, Slack, Speak, Wolfram, and Zapier.

Safety and Societal Implications

Connecting language models to external tools introduces significant risks, including the potential for harmful or unintended actions, sophisticated prompt injection, and the misuse of information.

Mitigation Strategies

OpenAI has implemented several safeguards based on internal and external red-teaming exercises:

  • Safety-by-Design: Restricting risky plugin behaviors and improving transparency regarding when plugins are operating.
  • Iterative Deployment: Gradually rolling out access to limit the blast radius of potential issues.
  • Evaluation Frameworks: Utilizing the open-sourced Evals framework for safety and capability evaluations.

Economic Impact

OpenAI references research indicating that language models with tool access will likely have a greater economic impact than those without, potentially accelerating job transformation, displacement, and creation.

Sources