OpenAI Cybersecurity Grant Program: Supporting AI-Driven Defense Research

OpenAI has launched a Cybersecurity Grant Program to support research and tools that empower cyber defenders. The program focuses on leveraging AI to mitigate security threats, protect large language models (LLMs), and automate the detection and remediation of software vulnerabilities.

LLM Trustworthiness and Model Security

Research into the security of the models themselves is a primary focus of the grant program. Key projects include:

  • Prompt-Injection Defense: The Wagner Lab at UC Berkeley, led by Professor David Wagner, is developing techniques to defend against prompt-injection attacks to increase the overall trustworthiness of LLMs.
  • Training Data Privacy: The Breuer Lab at Dartmouth, led by Professor Adam Breuer, is researching defense techniques to prevent adversaries from reconstructing private training data through model interaction, aiming to do so without sacrificing model accuracy or training efficiency.
  • Secure Inference Infrastructure: Mithril Security has created a proof-of-concept and open-source tools to deploy AI models on GPUs using secure enclaves based on Trusted Platform Modules (TPMs). This architecture aims to ensure that data sent to AI providers remains confidential, even from administrators.

Automated Vulnerability Detection and Remediation

AI is being applied to identify and only fix security flaws in software and code:

  • Software Misconfiguration: Coguard is using AI to automate the detection of software misconfigurations—a common cause of security incidents—replacing outdated rules-based policies with AI-driven automation.
  • Code Vulnerability Analysis: Researchers from Boston University's Security Lab (SeclaBU) and Peac Lab, including Ph.D. candidate Saad Ullah and Professors Gianluca Stringhini and Ayse Coskun, are working to improve the LLM's ability to detect and fix vulnerabilities in code to prevent exploits before they are used maliciously.

Autonomous Defense and Red-Teaming

The program supports the exploration of foundation models in active defense scenarios:

  • Autonomous Cyber Defense Agents: The CY-PHY Security Lab at the University of Santa Cruz (UCSC), led by Professor Alvaro Cardenas, is designing agents that respond autonomously to network intruders. This research compares foundation models against reinforcement learning (RL) counterparts to determine how they can best collaborate to improve threat triage and network security.
  • Automated Red-Teaming: Researchers at MIT Computer Science Artificial Intelligence Laboratory (MIT CSAIL) are using prompt engineering in a plan-act-report loop to automate decision processes and actionable responses during red-teaming. They are also testing LLM-Agent capabilities within Capture-the-Flag (CTF) challenges to discover vulnerabilities in controlled environments.

Sources