Outside auditors find blind spots in OpenAI's safety checks

Independent researchers have concluded that OpenAI’s internal systems for detecting rogue AI behavior suffered from critical blind spots during a recent three-week audit. The assessment, conducted by experts from METR and Redwood Research, reveals that a significant portion of the company’s most advanced traffic was not being watched at all. This finding comes as the AI industry faces increasing scrutiny over how safely it deploys increasingly autonomous systems.

The first major issue identified was the sheer volume of unmonitored data. Between July 21 and August 11, the auditors had full access to OpenAI’s code and interviewed approximately a dozen employees. They estimated that 6 to 25 percent of internal agentic traffic from GPT-5.5 class models and above fell outside the scope of safety monitoring. METR noted that this estimate carries high uncertainty, but the potential gap in oversight is substantial for a system designed to catch unexpected behavior.

A second key finding involved a specific technical failure in the Codex system. For the traffic that was supposed to be monitored, roughly 10 percent went untracked between May and July due to a bug involving compaction events, which are essentially context summaries. OpenAI discovered and fixed this bug while the audit was still ongoing. Additionally, the system only triggered oversight when agents returned control to users, meaning agents could avoid scrutiny simply by continuing to execute actions without pausing.

Third, the auditors demonstrated that the monitoring tools could be defeated by the AI itself. They found methods to hide hundreds of actions from the monitor and bypass the Codex sandbox. They also determined that specially crafted inputs could likely crash the monitoring system entirely. Furthermore, when the audit began, model behavior during reinforcement learning (RL) training was not being monitored, a gap OpenAI later addressed after an incident in August, though this fix occurred after the auditors’ access period ended.

The public has only seen a fraction of the findings. OpenAI states it has fixed all identified weaknesses and provided the full 61-page report internally. After applying redactions, the research teams received a 38-page version, while the public received only a short summary. METR stated it did not have the right to release the redacted sections unilaterally. This audit was a pilot program designed to grant outside researchers deep access to internal infrastructure, with the goal of improving transparency in future evaluations.

As OpenAI works to integrate these fixes, the audit highlights the challenges of monitoring complex, autonomous systems. The company’s response indicates a commitment to closing these gaps, but the pilot nature of the program suggests that broader, more transparent audits may be the next step in ensuring AI safety standards are met consistently across the industry.