Mistral Moderation API Release

Mistral AI has launched the Mistral Moderation API, a content moderation tool designed to provide system-level guardrails for AI deployments. This API is the same technology powering the moderation service in Le Chat, now made available for users to tailor to their specific application safety standards.

LLM-Based Classification and Capabilities

The Mistral Moderation API utilizes an LLM classifier trained to categorize text inputs into nine distinct policy categories. By leveraging a Large Language Model for classification, Mistral AI aims to provide a more scalable and robust moderation system compared to traditional methods.

Multilingual Support

The model is natively multilingual, with specific training on the following languages:

  • Arabic
  • Chinese
  • English
  • French
  • German
  • Italian
  • Japanese
  • Korean
  • Portuguese
  • Russian
  • Spanish

Input Endpoints

To address the fact that undesirable content is often context-dependent, the API provides two distinct endpoints:

  1. Raw Text Endpoint: For analyzing standalone text inputs.
  2. Conversational Content Endpoint: For analyzing text within a conversational context. In this case, the model is trained to classify the last message of the conversation.

Safety Policy and Guardrails

The Moderation API addresses model-generated harms, including the prevention of unqualified advice and the protection of Personally Identifiable Information (PII). The full set of policy definitions is available in Mistral AI's technical documentation.

Performance Metrics

Mistral AI has shared performance data based on AUC PR (Area Under the Precision-Recall Curve) across various policies using an internal testset. This metric is used to evaluate the effectiveness of the classifier across the nine defined policy categories.

Sources

Related

  • Dispatch
    Mistral AI Shieldstral 1.0 3B Release

    Mistral AI has released Shieldstral, a 3B open-weights multimodal safety classifier that uses a binary question-answering framework to enable policy-adaptive moderation without retraining.

  • Dispatch
    Mistral Shieldstral 1.0 3B Release

    Mistral AI has released Shieldstral 1.0 3B, an open-weights, policy-adaptive multimodal safety classifier that allows users to define moderation policies via natural language queries at inference time.

  • Dispatch
    Mistral AI Le Chat Release

    Mistral AI has released Le Chat, a multilingual conversational assistant that serves as a demonstration of Mistral models' capabilities for both individual users and business environments.

  • Dispatch
    Mistral AI La Plateforme Release

    Mistral AI has launched La Plateforme, a beta API service providing access to three generative chat endpoints and one embedding endpoint with varying performance and price tradeoffs.

  • Dispatch
    Mistral AI le Chat Update February 2025

    Mistral AI has unveiled an updated le Chat AI assistant featuring high-speed inference, integrated web search, code execution, and new Pro, Team, and Enterprise tiers.