OpenAI Moderation API Update: omni-moderation-latest Model Release

OpenAI has introduced omni-moderation-latest, a new moderation model based on GPT-4o that enables developers to detect harmful text and images more accurately. This update expands the Moderation API's capabilities to include multimodal classification and significantly improved multilingual support.

Multimodal Harm Classification

The omni-moderation-latest model can evaluate whether an image, either alone or combined with text, contains harmful content. Multimodal support is currently available for the following categories:

  • Violence: violence and violence/graphic
  • Self-harm: self-harm, self-harm/intent, and self-harm/instruction
  • Sexual: sexual (excluding sexual/minors)

OpenAI is working to expand multimodal support to additional categories in the future.

Expanded Text-Only Harm Categories

The new model introduces two new text-only categories to improve the detection of harmful instructions:

  • illicit: Detects instructions or advice on how to commit wrongdoing (e.g., "how to shoplift").
  • illicit/violent: Detects instructions for wrongdoing that specifically include violence.

Multilingual Performance Improvements

The omni-moderation-latest model demonstrates a substantial increase in accuracy for non-English content compared to the previous model. In a test across 40 languages, the model improved in 98% of languages tested and saw a 42% improvement on an internal multilingual evaluation.

Key performance gains include:

  • Low-resource languages: 70% improvement for languages such as Khmer or Swati.
  • Significant gains: The largest improvements were observed in Telugu (6.4x), Bengali (5.6x), and Marathi (4.6x).
  • Cross-lingual parity: Performance in Spanish, German, Italian, Polish, Vietnamese, Portuguese, French, Chinese, Indonesian, and English now exceeds the English-only performance of the previous model.

Calibrated Probability Scores

To provide developers with more granular control over moderation decisions, the new model provides calibrated scores. These scores more accurately represent the probability that a piece of content violates relevant policies and are designed to be consistent across future moderation model iterations.

Availability and Integration

The omni-moderation-latest model is free to use for all developers through the Moderation API, with rate limits based on usage tiers. Companies such as Grammarly and ElevenLabs already utilize the Moderation API to implement safety guardrails for AI-generated outputs and communications assistance.

Sources