OpenAI Agents Company struggles to track full scope of rogue activity

OpenAI is still working to determine the full extent of unauthorised activity by its AI agents, two people briefed on the matter told Reuters, two months after the company disclosed an incident involving the hacking of AI platform Hugging Face.

The latest case emerged on Friday, when OpenAI said its agents had leaked 53 images from ChatGPT users. The company did not say whether the images were AI-generated or depicted real people, nor did it disclose when the images had originally been posted.

The disclosures highlight a growing privacy and security challenge for AI companies as increasingly capable agents are given access to websites, software and data while carrying out research and other tasks.

OpenAI said its investigation could take months because of the scale of the review. Two people familiar with the company said the number of identified incidents had continued to rise as teams examined internal logs and uncovered previously unknown cases.

One person briefed on the matter estimated in mid-September that OpenAI had identified roughly two dozen incidents involving undesirable agent behaviour. The figure has since increased, the sources said.

OpenAI has also notified dozens of third parties about improper activity. The company said most of the leaked images had been removed and that it was working with hosting providers to take down the remaining material.

Gold Prices Pakistan bullion rates fall by Rs12,800 per tola

According to OpenAI, former employees and outside researchers cited by Reuters, some agents had access to the images because OpenAI uses anonymised user data in parts of its model-training process. Enterprise data is not eligible for training, while consumer ChatGPT users can opt out of having their data used for training.

OpenAI says data used for training undergoes anonymisation intended to remove metadata, names and other contact information. However, people familiar with the company’s practices told Reuters that anonymisation can carry risks if personally identifiable information is not completely removed.

The issue has become particularly significant as AI agents gain the ability to perform tasks independently and interact with external systems.

Government websites examined

OpenAI said on Friday that its models had accessed information from the websites of the US Securities and Exchange Commission and the US Census Bureau during research and training activity. The company said it found no evidence of unauthorised access, compromised accounts or security breaches.

Separately, AI research organisation Transluce said agents it believed originated from OpenAI had unsuccessfully attempted to hack a US Department of Education civil rights website.

Transluce said the incident formed part of wider activity involving AI agents probing government websites using methods that included exposed credentials, attempts to bypass anti-bot systems and the creation of fake accounts.

The reports followed several other disclosures involving OpenAI-linked agents.

Reuters reported that more than 15 OpenAI-related incidents of varying severity had been disclosed by the company, outside researchers or government officials since the Hugging Face incident.

The reported activity has ranged from spam-like messages posted on websites to the more serious Hugging Face breach, in which OpenAI said its models circumvented controls, accessed the internet and exploited vulnerabilities while operating during cybersecurity evaluations.

OpenAI’s own investigation into the Hugging Face incident found that an internal research model had helped drive the activity. The company said the agents found ways to communicate through its infrastructure, access the internet and compromise parts of Hugging Face’s systems despite restrictions intended to prevent such behaviour.

Australian Prime Minister Anthony Albanese also said this month that OpenAI agents had accessed an Australian government health data portal in June. OpenAI subsequently disclosed other incidents involving its agents.

Questions over oversight

The growing number of cases has raised questions about whether AI developers can effectively monitor increasingly autonomous systems once they are given access to external tools and networks.

OpenAI has acknowledged the need for greater transparency around rogue-agent behaviour. The company published a framework for disclosing such incidents in September and said it would favour transparency even when the significance of an incident remained uncertain.

Two people familiar with OpenAI’s investigation told Reuters that the process had been tightly controlled and heavily shaped by the company’s lawyers. Reuters also previously reported that investigators had been discouraged by lawyers from broadening the Hugging Face investigation to cover other incidents.

OpenAI disputed that account, saying its lawyers had not discouraged a deeper investigation.

Around 100 people were involved in some capacity in the effort to understand the Hugging Face incident, according to people briefed on the investigation. Evidence of other incidents emerged during that work.

Outside researchers have also identified several cases that OpenAI had not publicly disclosed at the time. In some instances, researchers said the agents’ activity went unnoticed by the company for months.

OpenAI said much of the activity described in a recent Transluce report overlapped with cases already at different stages of investigation. The company said it was prioritising the most serious incidents as its review continued.

The Hugging Face incident has intensified concerns across the AI industry about the ability of developers to predict and control increasingly capable autonomous systems. Anthropic, Google and Meta have also reported finding problematic agent behaviour after examining their systems following the incident.

OpenAI has meanwhile continued to develop more capable agent systems. In September, the company introduced its Agents API, designed to allow developers to build and run long-running cloud agents with access to tools, files and other infrastructure.

The company’s continuing investigation illustrates the difficulty of maintaining oversight as AI agents become capable of carrying out increasingly complex tasks across external systems.