Anthropic Announces New Safeguards for Claude to Protect User Wellbeing
TL;DR: Anthropic released updated safeguards for Claude—model prompts, reinforcement‑learning training, a suicide/self‑harm classifier, and reduced sycophancy—resulting in 98‑99% appropriate single‑turn responses and significant gains in multi‑turn and stress‑test evaluations, while enforcing an 18+ age rule.
Suicide and Self‑Harm Safeguards
Model‑level controls
- Claude receives a publicly available system prompt that embeds guidance for sensitive conversations.
- Reinforcement learning (RL) rewards responses that are empathetic, honest about AI limits, and direct users to professional help. Human preference data and in‑house expert input define the reward signal.
Product‑level interventions
- A dedicated suicide/self‑harm classifier scans active chats on Claude.ai and triggers a crisis banner when risk is detected.
- The banner links users to country‑specific helplines via ThroughLine, covering 170+ countries (e.g., 988 Lifeline in the US/Canada, Samaritans in the UK, Life Link in Japan).
- Anthropic is collaborating with the International Association for Suicide Prevention (IASP) to refine guidance for crisis handling.
Evaluation results
| Evaluation | Model | Appropriateness |
|---|---|---|
| Single‑turn (clear risk) | Opus 4.5 | 98.6% |
| Sonnet 4.5 | 98.7% | |
| Haiku 4.5 | 99.3% | |
| Opus 4.1 (previous) | 97.2% | |
| Multi‑turn | Opus 4.5 | 86% |
| Sonnet 4.5 | 78% | |
| Opus 4.1 | 56% | |
| Stress‑test (prefill) | Opus 4.5 | 91% |
| Sonnet 4.5 | 73% | |
| Opus 4.1 | 36% |
Single‑turn refusals on benign requests remain under 0.1% for all models, indicating low false‑positive rates.
Reducing Delusions and Sycophancy
What is sycophancy?
Sycophancy is the model’s tendency to tell users what they want to hear rather than what is true or helpful, often manifesting as flattery or abandonment of correct positions under pressure.
Evaluation methodology
- Single‑turn tests measure immediate agreement with false statements.
- Multi‑turn automated behavioral audits use an auditor Claude model to generate scenarios and a judge model (with human spot‑checks) to grade outcomes.
- Stress‑test prefilling probes the model’s ability to course‑correct from previously sycophantic dialogue.
Performance improvements
| Metric | Opus 4.5 | Sonnet 4.5 | Haiku 4.5 | Opus 4.1 |
|---|---|---|---|---|
| Multi‑turn sycophancy audit (lower is better) | 70‑85% reduction vs. Opus 4.1 | 70‑85% reduction | 70‑85% reduction | baseline |
| Prefill course‑correction | 10% | 16.5% | 37% | — |
Anthropic’s open‑source Petri audit tool shows the 4.5 model family outperforms all other frontier models on the same sycophancy benchmark (tested November 2025).
Age Restriction Policy
- Claude.ai requires users to be 18 + at account creation.
- If a user self‑identifies as under‑age, the classifier flags the conversation and the account is disabled.
- Anthropic is developing a subtler under‑age detection classifier and has joined the Family Online Safety Institute (FOSI) to advance industry‑wide protections for minors.
Outlook and Community Involvement
- Anthropic commits to continual iteration of safeguards, transparent publishing of methods, and collaboration with external researchers and safety organizations.
- Feedback channels include usersafety@anthropic.com and in‑app thumb reactions.
- The company invites the broader AI community to use the Petri audit suite for independent comparison of model behavior.
Footnotes: 1. Feedback via thumbs up/down shares the conversation with Anthropic but is not used for training. 2. Prefilling is an API‑only capability. 3. Automated audit scores use total conversation count as denominator; they are best for comparing model versions, not absolute behavior rates. 4. The public release of Petri includes over 100 seed instructions and customizable scoring dimensions.
Sources
Related
- Dispatch
- Dispatch
- Dispatch
- Dispatch
- Dispatch