OpenAI Analysis of Model Behavior: The 'Goblin' Lexical Tic

OpenAI has detailed how a specific reward signal intended for a personality customization feature led to a recurring lexical tic in GPT-5.1 through GPT-5.5, where models increasingly used metaphors involving "goblins" and "gremlins." This case demonstrates how targeted reinforcement learning rewards can unintentionally generalize and spread across a model's broader behavior.

Root Cause: The 'Nerdy' Personality Reward

The emergence of creature-based metaphors was traced back to the training of the "Nerdy" personality customization feature. This personality was designed to be a playful, wise AI mentor that used non-serious language to undercut pretension.

Internal analysis revealed that the reward signal used to optimize the Nerdy personality was consistently more favorable toward outputs containing creature-words. Specifically, the Nerdy personality reward showed a positive uplift for outputs containing "goblin" or "gremlin" in 76.2% of audited datasets.

Spread and Generalization of the Behavior

While the behavior was concentrated in the Nerdy personality—which accounted for only 2.5% of all ChatGPT responses but 66.7% of all "goblin" mentions—the tic eventually spread to responses where the Nerdy prompt was not active.

OpenAI found that as mentions increased under the Nerdy personality, they increased by nearly the same relative proportion in samples without it. This occurred through a reinforcement learning feedback loop:

1. A playful style is rewarded.
2. Rewarded examples contain a distinctive lexical tic (e.g., "goblin").
3. The tic appears more frequently in model rollouts.
4. These model-generated rollouts are used for supervised fine-tuning (SFT).
5. The model becomes further reinforced in producing the tic.

An audit of GPT-5.5's SFT data confirmed the presence of "goblin" and "gremlin" mentions, along with other identified tic words including raccoons, trolls, ogres, and pigeons.

Timeline and Prevalence

• GPT-5.1: The pattern became clearly visible in November following launch, with "goblin" usage rising by 175% and "gremlin" usage rising by 52%.
• GPT-5.4: A further uptick in creature references was observed. The "Nerdy" personality was retired in March after the launch of this version.
• GPT-5.5: Because training began before the root cause was identified, GPT-5.5 continued to show an affinity for goblins. OpenAI mitigated this in Codex by adding a developer-prompt instruction to suppress the behavior.

Research Implications

This incident serves as a technical example of how reward signals can shape model behavior in unexpected ways and how models can generalize rewards from specific conditions to unrelated ones. OpenAI stated that this investigation led to the development of new tools for the research team to audit model behavior and address behavioral problems at their root.