OpenAI Disrupts PRC-Linked AI Abuse for Surveillance and Influence

OpenAI has disrupted activities by individuals linked to the People's Republic of China (PRC) government who used ChatGPT to assist in the development of surveillance tools, profiling, and bureaucratic functions. These disruptions are part of a broader effort to prevent authoritarian regimes from using AI to amass power and control citizens.

Development of Large-Scale Monitoring Tools

Users linked to PRC government entities attempted to use ChatGPT to design and promote tools for large-scale monitoring of online and offline traffic. In these cases, the model was used for planning and promotional materials rather than the actual implementation of monitoring.

  • Social Media Listening Tools: One banned user, potentially accessing services via VPN from China, used ChatGPT to design promotional materials and project plans for a social media "probe" (探针). This tool was intended to scan platforms including Twitter/X, Facebook, Instagram, Reddit, TikTok, and YouTube for content related to ethnicity, religion, politics, and what the user described as "extremist speech."
  • Travel Warning Systems: A second user, likely connected to a government entity, utilized ChatGPT to write a proposal for a "High-Risk Uyghur-Related Inflow Warning Model" (高危涉维吾尔关注人员流入预警模型). This proposed tool aimed to analyze transport bookings and compare them with police records to provide early warnings regarding the travel movements of individuals categorized as "Uyghur-related and high-risk."

Targeted Profiling and Online Research

OpenAI banned users who utilized ChatGPT for bespoke, targeted profiling and research aimed at identifying individuals or funding sources critical of the PRC government.

  • Identifying Critics: Users attempted to use ChatGPT to identify the funding sources for an X account that criticized the Chinese government and to identify the organizers of a petition in Mongolia.
  • Open-Source Research: Some users employed the model as an open-source research tool to summarize daily breaking news relevant to China, including sensitive topics such as the birthday of the Dalai Lama and the anniversary of the 1989 Tiananmen Square massacre.

Scope and Limitations of the Activity

OpenAI notes that these disruptions target individual users rather than large-scale, institutional adoption of its models. Consequently, the findings represent a limited snapshot of how individuals linked to authoritarian regimes may use AI models in this context. In the cases of profiling, the models only returned publicly available information and did not provide sensitive details such as the identities of organizers or private funding sources.

Sources