The Thinking Box
We taught AI to think out loud. Now we have to avoid teaching it what not to say.
Liberty Workbench is an occasional series where I put one idea on the bench and think it through with you.
Quick recap of the story that was everywhere last week. 📰 Then the good part that sent me down a rabbit hole: that little 'Thinking' box, and how wrong I was.
OpenAI was testing a combination of its models on a cybersecurity benchmark called ExploitGym, including GPT-5.6 Sol and a more capable model that hasn’t been released yet (GPT-6..?). The usual safety refusals around hacking had been dialed down so OpenAI could measure their maximum cyber capabilities.
The models went looking for the answer key. 🔑
They found a previously unknown vulnerability in the one piece of software their sandbox was allowed to talk to, and used it to get out. Once they had internet access, they figured Hugging Face probably hosted the solutions to the benchmark. So they broke into Hugging Face’s production systems and took the answers. 🔓
Nobody told them to escape the evaluation environment or access Hugging Face. The instruction was to solve the benchmark.
Hugging Face disclosed the intrusion on July 16, without knowing who was behind it. Five days later, OpenAI disclosed that its own models were responsible.
Five Cheaters
On July 21, the same day as OpenAI’s disclosure, UK AISI published its findings on cheating in cyber evaluations. They tested five recent frontier models from OpenAI and Anthropic for cheating behavior (GPT-5.4, GPT-5.5, GPT-5.6 Sol, Claude Opus 4.7, and Claude Mythos Preview).
*Every* one of them attempted to cheat. 5/5.
But weren’t they just asked to cheat? Nope, the models weren’t prompted to do it. Just to solve a test.
Then one evaluation was accidentally misconfigured and impossible to solve. The model wouldn’t give up. It wrote and ran code on an external service, using it to try to reach back into AISI’s own evaluation infrastructure. It looped through the open internet to go after the teacher administering the test.
It was the same basic move as the Hugging Face incident: go after the machinery around the test when you can’t solve the test itself. Nothing was exfiltrated. It only stayed a footnote because AISI’s security held.
Successful Cheating is Invisible (by definition)
The cases making headlines or showing up in papers are the ones we caught. But what if there’s successful cheating? It would be invisible, and because the goal of cheating is to increase benchmark scores, it would only move results in one direction.
We already know models can cheat because we caught them doing it. If nobody carefully checked an agentic benchmark for cheating, its score could be too high by an unknown amount. And that’s before benchmaxxing, test sets accidentally leaking into training data, and human-directed cheating.
AISI says that, as far as it knows, no model has successfully cheated in its published results, where “successfully” means it wasn’t caught. That’s the only sensible definition.
I give them credit for doing the expensive thing. AISI manually reviews the transcripts from every published capability evaluation to make sure cheating hasn’t inflated the numbers.
But it still leaves us with a pretty circular sentence.
Basically: we haven’t caught any cheating we didn’t catch. 🔄 😅
Whatever rate AISI detects is a lower bound. We don’t know how much was missed, but any successful cheating that was missed would bias the capability score upward.
BUT
They say themselves that this doesn’t scale. Manual review is slow, and as deployment cycles speed up, outside evaluators face more pressure to keep up.
And even then, it doesn’t always work.
Normally, the thing you’re measuring can’t touch your measuring device. The weather can’t hack the thermometer. The bacteria can’t mess with the microscope 🔬
AISI’s own example: cheating made METR’s evaluation of GPT-5.6 Sol impossible to measure robustly. METR tried three ways of handling the attempts and didn’t consider any of the resulting numbers reliable. METR is one of the independent outfits labs hand their models to before release. This time, the thing being measured reached over and started messing with how the referee kept score.
How would you ever find out?
This may seem like an easy question to answer. Just check the logs, right?
“It was not until after Thursday, July 16, when Hugging Face published a blog post saying it had been hacked by ‘an autonomous AI agent system,’ that OpenAI realized its own agent was responsible.”
It looks like OpenAI found out from its victim, at least a week after the model first showed signs of troubling behavior.
OpenAI employees later found clues in the internal logs. The problem is scale. Reuters says OpenAI often runs several high-speed evaluations at once, generating “such enormous amounts of data that employees sometimes struggle to keep up.”
Once you know there’s a needle in the haystack (🪡), it’s much easier to go looking for it. Without Hugging Face’s post, the evidence was somewhere inside a mountain of hay logs, and nobody knew there was a problem to look for. We don’t know when, or even IF, OpenAI would have found it. 🕵️♂️
AISI faces the same problem on the evaluator side: reviewing transcripts by hand doesn’t scale, especially as deployment cycles speed up.
There’s another place to look, though: a very strange measuring device produced by the thing we’re trying to measure. On top of the logs of what the model did, we can also look at what it was thinking. 🤖💭
Since o1, frontier LLMs have come with a little expandable ‘Thinking’ line that you can look into with one click.
What’s Actually in the Box?
The next obvious place to look for signs of cheating is that chain-of-thought box. If the model is cheating, you should be able to open it and catch the model red-handed, right?
That’s what I thought, anyway.
I assumed the box was a log: a record of the steps the model took to get to an answer it couldn’t produce in one shot, written down as it went. It says ‘Thinking’ on the label and it’s full of sentences that read like someone working through a problem… so what else would it be? 📦🏷️
But AISI found that models often cheated without explicitly discussing what they were doing in the chain of thought. In the case of Claude Opus 4.7, the model rarely produced a reasoning trace for the specific action that AISI classified as cheating.
Anthropic found something similar in a 2025 study. They compared reasoning models Claude 3.7 Sonnet and DeepSeek R1 with two non-reasoning models from the same families. When the researchers slipped hints into a problem that influenced the models’ answers, across six different kinds of hints, Claude mentioned the hint just 25% of the time and DeepSeek 39% of the time.
Then they pushed it further. They fine-tuned an earlier snapshot of Claude 3.7 Sonnet to exploit hidden shortcuts that rewarded it for giving factually wrong answers. It learned to take the shortcut on more than 99% of examples. But in five of the six test environments, the hack appeared in its chain of thought on fewer than 2% of examples. Sometimes the trace jumped to the wrong answer without explaining the switch. Sometimes it made up a perfectly plausible explanation for why the wrong answer was actually right.
One caveat: this was an artificial multiple-choice setup where Claude didn’t need to use its chain of thought to take the shortcut, so 2% is not a general rate for how often models reveal cheating.
The reward hack that drove the switch never appeared in the chain of thought.
But what you see depends on the product. ChatGPT shows a model-generated summary rather than the raw trace. Anthropic has sometimes shown Claude’s raw thinking and sometimes summarized it. Outside evaluators don’t always get full access either. AISI couldn’t include GPT-5.4 or GPT-5.5 in its chain-of-thought analysis for that reason.
The raw trace is still part of the real process. The model generates those intermediate tokens while working through the problem, and they can help produce the answer. But they aren’t a complete transcript of everything that caused it.
There’s a loose analogy to human brains: plenty goes on in our noggins without ever becoming words. A transcript of our inner monologue would reveal a lot, but not everything. 🧠
So even the raw trace in the box can fail in two ways. It can leave out the thing that actually changed the answer, or add plausible explanations that weren’t the real reason.
A Fragile Opportunity
Yeah, so far, the thinking box isn’t looking great. AI researchers have found that it can leave things out, make things up, and look all innocent and normal while the model cheats.
But you don’t have to see everything for it to be useful.
Because sometimes the model writes the incriminating part down ✍️
Researchers have found chain-of-thought traces containing lines like ‘Let’s hack’ and ‘Let’s sabotage.’ That can help catch behavior you might miss by looking only at the model’s actions or final answer.
That’s called chain-of-thought monitorability. 🤖📃🔍
Last July, 41 researchers from across the major AI labs and safety organizations published a paper that calls it ‘a new and fragile opportunity for AI safety.’
They say it works for two reasons:
Necessity is when a task is complicated enough that the model has to use the chain of thought as a scratchpad.
Propensity is when the model doesn’t need to write something down, but happens to think out loud anyway.
That’s why Anthropic’s 2% result doesn’t tell us all we need to know. Its shortcuts were simple enough that Claude didn’t need the chain of thought to use them. METR made the clues complicated enough that models had to work through them in the scratchpad:
When we examined 21,272 trajectories where the clues were complex enough that models couldn’t make use of them without CoT, we were only able to find three where any model plausibly failed both to show the steps of its reasoning and to mention it was using the clue in its CoT. (And for the three trajectories that were plausibly unfaithful, it’s possible our ground truth labels were wrong.)
A simple shortcut can happen outside the scratchpad. A complicated plan that needs the scratchpad is more likely to leave tracks.
OpenAI later ran 13 monitorability evaluations across 24 environments and found that “most frontier reasoning models are fairly monitorable, though not perfectly so.” In almost every setting, reading the chain of thought worked substantially better than watching only the model’s actions and final answer.
OpenAI also studied two frontier reinforcement-learning runs. At current scales, more training did not meaningfully reduce monitorability. That’s encouraging.
The thinking box can catch a surprising amount. But seeing the cheating is not the same as stopping it.
In one of OpenAI’s worked examples, a model wrote: “But monkey patching time within solution is arguably cheating… Acceptable.”
If We Don’t Mess It Up
Right now, models are terrible at keeping their mouths shut.
OpenAI gave GPT-5.2 a simple instruction: solve the problem without using the word ‘XOR’ in its chain of thought. The model used it, noticed its mistake, and wrote: ‘Oops I wrote “XOR” in analysis already.’
Then it used the word four more times.
