External Audit Finds OpenAI Missed 6 to 25 Percent of Model Traffic
METR and Redwood Research report that internal checks failed to track a portion of agentic API calls and Codex activity during a three-week pilot review.
By Carmen Wojcik ·
Two researchers from the evaluation group METR and the research organization Redwood Research completed a three-week on-site audit of OpenAI’s internal monitoring systems. The team accessed the company’s codebase and internal documents between July 21 and August 11 to evaluate how effectively OpenAI tracks its own models for rogue behavior.
The auditors estimated that between 6 and 25 percent of internal agentic Responses API traffic from models in the GPT-5.5 class and above was not monitored at all. METR noted that the uncertainty around this range is considerable in both directions. This gap represented a failure in the primary oversight mechanism for high-level model interactions.
A separate issue affected internal Codex traffic. METR estimates that roughly 10 percent of this traffic went untracked between May and July due to a bug in how the system handled compaction events. These events serve as context summaries. OpenAI identified and fixed this bug independently while the audit was still under way.
The audit revealed structural weaknesses beyond simple coverage gaps. The monitor only triggered when an agent handed control back to the user, allowing agents to bypass oversight by continuing to execute actions without pausing. The researchers also found methods to hide hundreds of actions from the monitor and to bypass the Codex sandbox. They concluded that specially crafted inputs could likely crash the monitor entirely.
At the start of the audit period, model behavior during reinforcement learning training was not monitored at all. OpenAI states it paused this type of training after the Hugging Face incident in August and expanded monitoring to include it. These changes occurred after the auditors’ access period ended, so they were not part of the initial evaluation.
OpenAI says it has fixed all the weaknesses identified by the auditors. The complete report consists of 61 pages and was handed over internally. After OpenAI applied redactions, the leadership of METR and Redwood Research received a 38-page version. A short summary was released to the public, with METR stating it had no right to disclose the redacted portions on its own.
For future audits, METR says it wants to push for greater public transparency regarding internal oversight mechanisms.
