AI & ComputeLOT 261122 · 4 min read

OpenAI Cuts Reasoning-Extraction Attack Success to Under 1%

OpenAI says reasoning-extraction attacks now succeed 0.2% of the time on GPT-6.2 Astra, down from 54.0% on GPT-6 Astra, and it bans tens of thousands of accounts weekly over distillation.

By Tom Whitfield722 words

Key takeaways

  1. Reasoning-extraction attacks on earlier requests fell from 54.0% success on GPT-6 Astra to 0.2% on GPT-6.2 Astra in OpenAI's red-teaming test.
  2. Attacks on the current request's reasoning fell from 22.9% to 0.9%, aggregated across six attacker models.
  3. Weekly enforcement against distillation campaigns has ranged from thousands to tens of thousands of accounts, rising this summer.
  4. OpenAI named DeepSeek (February congressional briefing) and individuals associated with Moonshot AI (September attribution).
  5. OpenAI backs the Chip Security Act, the AI OVERWATCH Act and the MATCH Act in Congress.
OpenAI says it has cut attacks that extract its models' hidden reasoning to under 1%
OpenAI says it has cut attacks that extract its models' hidden reasoning to under 1% — blickpixel / pixabay

OpenAI said on Friday, Nov. 20 that attacks attempting to extract its models' hidden reasoning now succeed less than 1% of the time, down from as much as 54.0%, in its most detailed account yet of how it fights adversarial distillation — the unauthorized use of a frontier model's outputs to train a copy.

In an internal adaptive red-teaming test, attacks that tried to recover protected reasoning from an earlier request succeeded 0.2% of the time against GPT-6.2 Astra with its updated safeguards. The same attack succeeded 54.0% of the time against GPT-6 Astra with its original protections. Attacks on the reasoning of the current request fell from 22.9% to 0.9% in the same black-box test, aggregated across six attacker models. OpenAI said it is now applying selected improvements to GPT-6 Astra as well.

What is distillation, and why does OpenAI treat it as a safety issue?

Distillation is a standard machine-learning technique in which a "student" model learns from data produced by a "teacher" model — its answers, its reasoning, or its judgments about which answers are better. OpenAI argues the stakes go beyond intellectual property. An unauthorized copy can inherit a model's capabilities without its safeguards, so it may help with cyberattacks or biology work that the original model would refuse.

The company places particular weight on its models' chain-of-thought: the intermediate steps a model generates before answering. It keeps those steps hidden and shows users only summaries. In an internal study, OpenAI fine-tuned the same pretrained student on outputs from Astra-level teachers working on coding and defensive security tasks. After supervised fine-tuning, the student trained on exact reasoning traces generally beat the one trained on summaries. After a round of reinforcement learning, the exact-trace student still scored higher at the largest inference budget tested on ExploitBench and SEC-bench Pro, while the summary-trained student did better at lower budgets. That gap, OpenAI says, is why it guards the raw traces.

What changed in the defenses?

OpenAI described two levels of defense. For individual requests, it trains models to recognize and refuse extraction attempts, runs input classifiers to spot those attempts, and runs output classifiers to catch leaks in both messages and tool-call arguments. It also restricts, at the API level, how encrypted reasoning can be reused.

With GPT-6.2 Astra and Sol, which launched this week, protected reasoning is tied to the conversation that produced it. If an earlier message is rewritten, the API drops the reasoning that followed it, and older model families cannot read the new models' reasoning. OpenAI said it also closed a pathway that could have exposed another user's encrypted reasoning.

At the campaign level, OpenAI said it has seen multiple unauthorized distillation campaigns on its platform this year. It investigates the account networks behind them and shares indicators of compromise with other model providers. Weekly enforcement has ranged from thousands to tens of thousands of accounts, with the rate rising significantly this past summer. The company also announced this week that resellers of access to its models now need its approval, citing a consistent-standards policy for providers offering its models.

Who does OpenAI name, and what is it asking for?

OpenAI said it briefed Congress in February on activity indicating DeepSeek was attempting to distill US frontier models. In September, it attributed a core part of a reasoning-extraction campaign to individuals associated with Moonshot AI, the developer of Kimi. It said it plans to share more about recently disrupted campaigns. Reuters reported on Nov. 17 that Chinese companies are using answers from US models to train domestic rivals as chip curbs tighten.

On policy, OpenAI said it supports legislation giving the US government more tools to identify and sanction foreign actors that illicitly extract capabilities from American models. It has told Congress it backs the Chip Security Act, the AI OVERWATCH Act and the MATCH Act. Anthropic has described a similar fight and has also banned tens of thousands of accounts, The Decoder noted.

The red-teaming numbers show a sharp drop in one lab's own tests; whether the safeguards hold against sustained, well-funded campaigns is the question OpenAI's promised disclosures on recent disruptions will help answer.

mlq.ai (Source)

Filed under

Share this article:

More from Tom Whitfield

Tom Whitfield

Show full bio

Staff writer covering companies, deals and strategy in Semiconductors, chip manufacturing, hardware, AI compute at Chip Dispatch.

1,200 articles

Read also

« Previous article