Top AI companies have spent plenty of time warning that models could make it easier for someone to develop a biological weapon. Now, Anthropic says it actually had to cut off a group of researchers after their behavior raised alarms.
According to "The New York Times", Anthropic blocked several users this year after spotting activity that looked suspiciously close to bioweapons research.
To be clear: Anthropic doesn't actually know if these researchers were trying to develop a deadly pathogen.
Advanced biology is inherently "dual use" — the same research that can lead to a lifesaving vaccine can potentially be used for something dangerous. But Anthropic flagged the behavior. The researchers reportedly circumvented geographic restrictions to access Claude and took steps to obscure the purpose of their work. That pattern was enough for Anthropic to pull the plug.
How they got around Claude's safeguards
The researchers didn't exactly "hack" Claude. Instead, Anthropic says they found ways to conceal both where they were accessing it from and what they were ultimately trying to accomplish.
In one case, a researcher reportedly accessed Claude from an unsupported region through U.S.-based virtual infrastructure and used an automatically generated account. Anthropic also found Claude requests being routed through a platform serving numerous life-sciences researchers, including virologists with ties to civilian and military institutions.
The researchers also attempted to obscure the purpose of their work to avoid triggering Claude's safeguards. That's particularly difficult to detect in biology because an individual request can look perfectly legitimate even when it contributes to a potentially dangerous larger project.
One of the cases reportedly involved a scientist seeking Claude's help drafting a grant proposal for research that would genetically modify the chikungunya virus to study characteristics including transmissibility and immune evasion. That's research that can have legitimate scientific purposes, which is precisely why Anthropic says it couldn't determine whether the researchers' ultimate intentions were benign or malicious.
Anthropic nevertheless decided the combination of sensitive research and attempts to conceal the activity was enough to shut down the accounts.
Why biology is the ultimate stress test
An AI can't mix chemicals in a petri dish or create bioweapons with physical lab equipment. But AI can act as an accelator.
For someone who already knows their way around a lab, a capable model can break down complex research, analyze data, troubleshoot problems and bridge niche technical gaps. It can give humans everything they need to make dangerous work easier and faster.
That leaves AI companies facing a massive gray area. A legitimate scientist and someone pursuing a biological weapon could ask many of the same questions. And misuse doesn't necessarily arrive as one blatant prompt. Someone can divide a larger project into dozens of innocent-looking tasks, making it much harder for a safety system to understand the bigger picture.
Then there's the problem of false positives. Make those safeguards too aggressive and you risk blocking legitimate scientific research. Make them too permissive and potentially dangerous activity can slip through.
It all comes down to human intent
It's important to clarify what didn't happen here, especially after Jacob Coxon's resignation from Anthropic yesterday, concerns about AI are at an all time high. We worry about models going rogue or autonomous agents acting on their own, but Claude did not go rogue, and autonomous AI agents didn't start plotting a bioweapon on their own. Humans were driving the bus.
That makes the risk far more immediate. The concern in this case isn't a super intelligent AI suddenly deciding to harm humanity. It's that powerful models are giving people capabilities they might not otherwise have had.
For everyday users asking Claude to draft emails, summarize documents or write code, nothing changes. But for the industry, the game has shifted.
So, what can AI companies actually do?
There isn't a perfect fix here. Blocking obvious questions only works when someone is clear about what they are trying to do. If a user breaks a dangerous project into dozens of harmless-looking requests, AI companies need safeguards that can recognize the larger pattern rather than judge every prompt in isolation.
That could mean looking for repeated attempts to bypass a model's safety restrictions, monitoring suspicious patterns across interactions and putting tighter controls around the most powerful scientific capabilities. Companies can also test new models before release to determine how much they could help someone with little expertise perform potentially dangerous biological work.
But every additional safeguard comes with a trade-off. The same systems trying to spot someone developing a biological weapon could also flag a researcher doing legitimate work. That's why Anthropic's disclosure is so important. The next phase of AI safety is about deciding when a pattern of otherwise legitimate-looking behavior becomes dangerous enough for a company to intervene.
Follow Tom's Guide on Google News and add us as a preferred source to get our up-to-date news, analysis, and reviews in your feeds. Subscribe to Tom's Guide on YouTube and follow us on TikTok.
More from Tom's Guide
