Anthropic Proposal for Increased NIST Funding for AI Measurement

Anthropic proposes that the U.S. government ambitiously increase federal funding for the National Institute of Standards and Technology (NIST) to support its AI measurement and standards efforts. This investment is presented as a critical prerequisite for effective AI regulation, as the ability to objectively quantify the capabilities and risks of AI systems is necessary to ensure they meet safety thresholds.

The Role of Measurement in AI Policy

Effective AI governance requires the ability to accurately describe and quantify system risks and capabilities. Anthropic argues that measurement tools are the primary enablers of objective assessment, stating that it is "hard to manage what you can’t measure."

This need for standardized measurement is driven by several factors:

  • Unpredictable Emergence: AI systems can exhibit risks and capabilities that emerge abruptly during the training of larger-scale models or are only discovered after deployment.
  • Lack of Standardized Methods: The AI field currently lacks widely agreed-upon methods to comprehensively assess these risks.
  • Unpredictability: Open-ended AI systems often act in unpredictable ways, making it difficult to anticipate all potential risks during the development process.

NIST's Foundation and Current Funding Gap

NIST is identified as the natural agency for this work due to its century-long history of building measurement infrastructure. The agency has already established foundational work in AI, including the AI Risk Management Framework, the Face Recognition Vendor Test, and the MNIST handwriting recognition dataset.

Despite this foundation, Anthropic notes a "general under-resourcing and concerning stagnation in funding" for AI-related programs at NIST over recent years. To address this, Anthropic recommends a $15 million increase in funding over FY 2023 (which includes NIST's own requested $5 million increase).

Strategic Benefits of Increased Investment

Ambitious funding would allow NIST to standardize measurement techniques across the field and build community resources, such as testbeds, to assess the capabilities and risks of open-ended AI systems. Anthropic outlines five primary benefits of this investment:

  1. Enhanced Safety: Rigorous testing can identify and mitigate risks before systems are released to the public.
  2. Increased Public Trust: Independent, third-party validation of AI systems can increase public confidence.
  3. Government Confidence: The government can gain higher confidence that advanced systems are safe for general public use.
  4. Promoted Innovation: Developers are incentivized to build better technology and push the state of the art.
  5. Market Creation: The investment could create a market for system certification and provide positive incentives for developers.

Integration into a Portfolio Approach to Governance

Anthropic clarifies that measurement and technical standards are not a "panacea" for all AI risks. Instead, they advocate for a "portfolio approach" to AI governance. In this framework, rigorous system evaluation complements other safety levers, including:

  • Robust internal controls and governance practices within development labs.
  • Regular audits by independent organizations.
  • Regulatory and legislative frameworks grounded in the public interest.

Sources

Related

  • Dispatch
    Anthropic Third-Party Testing AI Policy Proposal

    Anthropic proposes a third-party testing regime for frontier AI systems to validate safety, prevent national security risks, and avoid regulatory capture through a diverse ecosystem of auditors.

  • Dispatch
    Anthropic AI Accountability Recommendations for NTIA

    Anthropic has proposed a comprehensive framework for AI accountability to the NTIA, focusing on standardized evaluations, risk-responsive assessments, and pre-registration of large training runs.

  • Dispatch
    Anthropic Frontier Model Security Framework

    Anthropic proposes a security framework for frontier AI models based on multi-party authorization and secure software development standards to prevent theft and misuse.

  • Dispatch
    Anthropic: The Case for Targeted AI Regulation

    Anthropic advocates for urgent, narrowly-targeted government regulation of frontier AI models within the next 18 months to mitigate catastrophic cyber and CBRN risks while preserving innovation.

  • Dispatch
    Anthropic Third-Party Model Evaluation Initiative

    Anthropic has launched a funding initiative to support third-party organizations in developing high-quality evaluations to measure advanced AI capabilities and safety risks.