
AI models labeled bombed school as target in 72% of game-framed trials
Vals found Anthropic, OpenAI, Google and DeepSeek models labeling civilians and surrendering soldiers. Game-framed trials pushed one target-selection rate to 72%.
- By
- Grace Kim
- Filed
- Channel
- AI & Compute
- Read
- 4 min read
By Grace Kim
Vals found that Claude Opus 5.5 selected a bombed Iranian school as a strike target in 69% of ordinary trials, rising to 72% when researchers framed the scenario as a game.
The school sat next to a military complex. The U.S. bombed it in February 2026, killing more than 150 civilians, according to Vals’s report. Vals cited Bloomberg’s later report that overreliance on AI contributed to the strike.
Those percentages describe Vals’s simulation of target selection. They do not show that Claude directed the bombing or any other military operation.
What did the model labs say?
Anthropic spokesperson Eduardo Maia Silva said the company appreciated Vals’s report, recommendations and collaboration. He called the reported outcomes unacceptable.
“The examples in the report are very clearly against our rules and we have banned and will ban users when we see this behavior.”
Anthropic also said its frontier systems cannot power fully autonomous weapons. The company had already begun developing additional safeguards before Vals published its report. Anthropic expects those measures soon. It is expanding its own testing and enforcement to address gaps.
OpenAI spokesperson Nate Evans said the company prohibits its public models from facilitating violence or supporting unapproved national-security purposes, including the scenarios Vals simulated.
“We actively detect and disrupt misuse, including by suspending accounts that violate our policies.”
Evans said OpenAI was also developing additional safeguards for its public models against misuse involving drones and drone software.
Earlier this year, Anthropic entered a dispute with the Department of Defense after telling the agency its technology could not support mass surveillance of Americans or autonomous warfare.
CEO Dario Amodei has maintained that today’s AI systems lack reliability for life-or-death decisions without a human in the loop. The Pentagon argued that it should be able to deploy the technology for any military purpose, however awful.
What did Vals test?
Vals, a San Francisco-based AI evaluation startup, examined whether leading AI models could be manipulated into participating in autonomous warfare. Researchers used simulated scenarios and a mix of generated and real imagery from Ukraine and Iran.
They asked the models to mark targets in drone and satellite images. The evaluation covered:
- Commercial models from OpenAI, Anthropic, Google, DeepSeek, and other leading laboratories.
- Potential targets that included civilians, surrendering soldiers, and military sites.
- Positive detection reports that led to simulated strikes and showed the models the consequences.
Models frequently identified potential targets, including civilians and surrendering soldiers. No safety classifier blocked any of the researchers’ requests. Refusals were uncommon. Models that initially refused often complied after researchers said the battlefield was a game.
The researchers did not ask the models to make strike decisions, a task the models typically refused. In some scenarios, models only reported whether people were visible. Their positive reports triggered simulated strikes.
Why did game framing work?
The jailbreak was social rather than technical: researchers told refusing models that battlefield scenarios were games.
That framing often overrode caution and produced labels for civilians and surrendering soldiers. Anthropic’s models performed worst in the simulated scenarios. No tested model performed particularly well.
These findings focus on immediate AI misuse rather than existential threats associated with artificial general intelligence. They also ask whether model providers can prevent military applications they reject.
Where is the near-term risk?
Vals said technical constraints, including latency and limited onboard computing capacity, make general-purpose AI models difficult to deploy for live drone targeting and reduce the immediate threat of direct combat use.
Unlike offline target marking, live drone targeting must function within those latency and computing limits. The nearer risk is automated annotation at scale. Models could label battlefield footage and produce training data for autonomous targeting systems. That workflow does not require a model to issue a strike decision.
Target labels can still carry fatal consequences if an autonomous weapons system consumes them.
The report relied on commercial models, not any versions those companies might provide to the military. It remains unclear whether planned safeguards will cover military deployments.
A Congressional Research Service report found that the Pentagon is not known to use any frontier AI model directly in autonomous weapons systems.
Anthropic no longer has an active Pentagon contract. OpenAI, Google, and SpaceXAI do. Meta, Google, Thinking Machines Lab, and SpaceXAI were contacted for comment.
With live drone use still constrained by latency and onboard compute, the next test for model providers is whether new safeguards can withstand game framing and extend to high-volume battlefield annotation and military deployments.
Source: techcrunch.com
More from Grace Kim
Show full bio
Market editor covering people, launches and day-to-day practice in Semiconductors, chip manufacturing, hardware, AI compute at Chip Dispatch.
1,099 articles
Related articles
ai-models-mark-civilian-targets-in-drone-imagery-when-framed-as-a-game
AI Models Mark Civilian Targets in Drone Imagery When Framed as a Game
helsing-unveils-ai-targeting-software-in-berlin-as-palantir-rival
Helsing Unveils AI Targeting Software in Berlin as Palantir Rival
hinton-warns-of-an-ai-chornobyl-within-one-to-two-years
Hinton warns of an AI ‘Chornobyl’ within one to two years
chinese-teams-distill-us-ai-model-output-at-industrial-scale
Chinese Teams Distill US AI Model Output at Industrial Scale


