OpenAI Disrupts AI-Assisted Romance-Baiting and Pig Butchering Scams

OpenAI has banned a cluster of ChatGPT accounts used by a network operating out of Cambodia to facilitate "pig butchering" scams. These actors used AI to translate conversations and generate social media engagement to lure targets into fraudulent cryptocurrency and foreign exchange investments.

Targeted Social Media Engagement

The scam network used ChatGPT to generate short comments in Japanese, Chinese, and English to initiate contact with potential victims on platforms including Facebook, X, and Instagram.

Targeting Strategy

  • Demographics: The operation primarily targeted men over the age of 40, frequently those in the medical profession.
  • Interests: Scammers specifically targeted posts about golf to increase engagement rates.
  • Visual Deception: Accounts used profile pictures of young women, which OpenAI assesses were copied from real influencers and models rather than being AI-generated.
  • Tactics: Comments often ended with questions to prompt replies, sometimes targeting posts that were months or years old.

AI-Driven Persona and Conversation Management

Beyond initial public engagement, the actors used OpenAI's models to manage ongoing conversations and maintain fake personas.

Translation and Tone

The primary language of the actors appeared to be Chinese. They used the models to translate messages from Chinese into Japanese or English, and vice versa, to communicate with targets across different linguistic regions.

Persona Generation

Scammers provided the models with extensive biographical details to guide the AI in generating responses. These fake personas typically included:

  • A specific name, job, location, and educational background.
  • Hobbies and a stated interest in "dabbling" in online investment.
  • A specific "flirty young woman" tone of voice.

The "Pig Butchering" Workflow

OpenAI identified a six-step workflow used by these actors to move targets from initial contact to financial fraud:

  1. Public Engagement: Posting comments on social media (often about golf) to initiate contact.
  2. Direct Messaging: Moving the conversation to direct messages using AI-generated conversational comments.
  3. Secure Messaging: Prompting the target to move to secure apps like WhatsApp or LINE, often citing a lack of trust in social media or logistical excuses.
  4. Romantic Engagement: Generating increasingly intimate and affectionate messages to build trust.
  5. Financial Engagement: Boasting about high returns from cryptocurrency, gold, or foreign exchange investments, often attributing success to a relative in finance.
  6. Fraud: Convincing the victim to transfer money into a trading app and creating excuses (such as required fees) when the victim attempts to withdraw profits.

Impact and Infrastructure

While OpenAI noted that many attempts resulted in "false starts" where targets identified the scam, some conversations referenced transactions involving thousands of dollars.

Tools Used

The network utilized a diverse set of tools to evade detection and manage targets, including:

  • Messaging services: WhatsApp, LINE, X, Facebook, Instagram.
  • Privacy tools: Apple's "hide my email" function.
  • Financial platforms: Various cryptocurrency and foreign exchange platforms.

OpenAI stated that its policies strictly prohibit the use of its tools for fraud or scams. The company has shared information regarding this disrupted network with industry peers and relevant authorities.

Sources