Over the weekend, the US military reportedly rushed aircraft into the air and held armed troops on standby to board a Chinese ship.

A chatbot had falsely linked its cargo to a nuclear weapons program.

The operation was stopped, but only after officials bothered to take a closer look at the intelligence report that influenced some of the most powerful decision makers on the planet.

One source described the report as “entirely false” but nevertheless “almost started a war”.

The reported sequence of events that unfolded inside the world’s most powerful military is extraordinary to say the least.

Well, not if you’re one of those AI campaigners who have been warning about this exact scenario for years.

Over the weekend, the American military reportedly rushed aircraft into the air and held armed troops on standby to board a Chinese cargo ship after a chatbot falsely linked its cargo to a nuclear weapons program.

MORE: Aus AI boom putting your home at risk

According to a CNN report citing four internal sources, a special operations analyst fed intelligence about the vessel’s manifest into a chatbot.

The system then combined open-source material with secret signals intelligence and produced the false conclusion that the ship was carrying components connected to a nuclear weapons program.

Want more finance news? Download our new app for even more coverage

The analyst then used AI again to package the finding into a neat intelligence report, polishing-up fundamentally unreliable material into authoritative language and then up the chain of command.

“The internal tools are mostly just copies of the commercial stuff wearing lipstick,” a former senior US official familiar with military and intelligence AI systems said.

The report spread, immediately sparking plans to intercept the ship. Armed personnel were reportedly prepared to board with military aircraft circling overhead before somebody checked the details were actually watertight.

This nauseating little scene is just one of potentially billions of catastrophic incidents that could arise as humanity leans on artificial intelligence more and more by the day.

The report spread, immediately sparking plans to intercept the ship. Armed personnel were reportedly prepared to board with military aircraft circling overhead before somebody checked the foundations were actually watertight. (Photo by Navy Office of Information / AFP)

The scenario is just one of many problems that inherently emerge from rapid integration, according to the hundreds of computer scientists who have been shouting to the hills about AI since its most primitive iterations.

Only now, after a particularly bizarre case of autonomous hacking followed by frazzled whistleblowers lifting the curtain, has the issue become undeniable.

The latest, potentially devastating, military slip-up has shone light on the tangible risks.

It has provided concrete evidence for those calling immediate regulation, as opposed to yet another vague warning about an all-powerful machine waking up and “optimising us out of existence”.

In this specific case, AI did not need consciousness, or even the control of a bomb. It only really needed to be confidently wrong in a format people were inclined to believe.

Nevertheless, its enormous potential is what keeps the world’s most cashed-up corporations and governments hooked.

Hegseth wants to ‘unleash experimentation’ in AI war

AI proponents say there’s simply no choice but so soldier ahead and address the risks as they emerge, because the risk of letting adversaries peel away would guarantee a bad outcome. That line of thinking is currently being put under immense strain, despite US President Donald Trump’s insistence that the negative press is a “hoax”.

In January, US Defence Secretary Pete Hegseth announced an “Artificial Intelligence Acceleration Strategy” intended to transform the Pentagon into an “AI-first” fighting force.

“We will unleash experimentation, eliminate bureaucratic barriers, focus our investments and demonstrate the execution approach needed to ensure we lead in military AI,” Hegseth said.

The accompanying strategy promised to spread advanced models throughout the military, “democratising AI experimentation and transformation across the Department by putting America’s world-leading AI models directly in the hands of our three million civilian and military personnel, at all classification levels”.

That policy deliberately rewards speed, aggressive experimentation and mass adoption. It is the backdrop of the rapidly evolving game of “blink first” playing out between the US and China.

Hegseth declares that “AI is only as good as the data that it receives, and we’re going to make sure that it’s there”. But officials familiar with the system say different internal agencies are using different models under inconsistent standards, with no common process for verifying their output.

US Secretary of War Pete Hegseth. (Photo by Finn Gomez / GETTY IMAGES NORTH AMERICA / Getty Images via AFP)

Naturally, enemies of the US are also using large language models in attempt to get ahead in the current Middle East spillover.

Just last week, AI giant Anthropic revealed an Iran-linked operator used its popular artificial intelligence model Claude to help track US warships and prepare targeting recommendations.

The “Iran-nexus threat actor” compiled targeting handbooks using publicly available information, including ship and aircraft transponder identifiers, commercial satellite-imagery tools and the names of US personnel scraped from military photographs.

The actor also directed Claude to research known vulnerabilities in maritime communications and industrial-control equipment, the company said in its September threat report.

The Pentagon has also clashed with Anthropic over the company’s refusal to remove safeguards intended to stop its Claude models being used for autonomous weapons and mass surveillance.

Anthropic, despite being on the “frontier” of development, said its systems were simply not reliable enough to be trusted with war.

The reported near-miss exposes the weakness in the promise that human oversight will make it safe. In this case, multiple humans duped by a convincingly authoritative source almost started another unnecessary geopolitical flashpoint.

Accelerating development has been the philosophy that has driven the US economic machine to its current heights. Only now are the broader conceptual risks being given the attention campaigners have been begging for.

Critics, including veteran campaigner and computer scientist Roman Yampolskiy, say the notion of attempting to regulate extremely capable AI is a dangerous misnomer, because it assumes humans will always understand — and have the collective agency to act on — the problems that need to be fixed.

At the rate of improvement and the speed at which the current “frontier” models run, campaigners like Yampolskiy say we are fast approaching a point where it could be impossible to individually parse through each and every mechanical thought process to check it aligns.

Treasury Secretary Scott Bessent said the two powers needed more transparency around failures carrying national-security consequences. (Brendan Smialowski/Pool Photo via AP, File)

‘A human rubber stamp’

AI and its use in war has been one of the main issues driving slowdown campaigners.

In 2019, former Pentagon official Paul Scharre warned that “overtrusting in machines could lead to accidents and miscalculation, even before a war begins”.

The charmingly-named Stop Killer Robots coalition, formed in 2012, now includes more than 300 organisations calling for meaningful human control over the use of force.

Human Rights Watch warned that same year that formal human oversight could become ineffective as militaries moved towards increasingly automated systems.

Their concerns were often reduced to a cartoon about Terminators roaming battlefields, but the more immediate problem was always much more dull.

In reality, automation bias, compressed decision times and exhausted humans accepting machine recommendations because the “computer appeared faster”, are proving just as potentially dangerous.

The Chinese ship incident shows how the idea of “human control” can become an illusion in itself if the data that informs decision making is entirely made up.

The Red Cross later outlined an almost eerily similar scenario: a human operator “launches an attack based solely on an AI-DSS’s output”, effectively becoming “a human rubber stamp”.

‘Approval button does not demonstrate control’

Over the past fortnight, warnings that were once treated as fringe doomsaying have dominated mastheads around the world.

But longtime AI researcher Nate Soares says the current panic has actually come as a relief for safety campaigners, because the things he and his peers have been harping on about for years were finally being discussed at scale.

In his book If Anyone Builds It, Everyone Dies, Soares lays out the tangible risks, but also attempts to explain the ineffable concept of runaway intelligence and the unknowns it will inherently bring.

Following the Hugging Face hacking event, the concept of AI “swarms” breaking free from training environments and rolling through the internet unchecked became realised.

Even the CEOs appeared to be spooked.

Anthropic chief Dario Amodei, OpenAI boss Sam Altman and Elon Musk immediately backed some form of slowdown at the frontier of AI development shortly after.

Their motives and their own role in creating the problem have also come under scrutiny, namely from Big Short investor Michael Burry.

The argument is that these are companies racing for money, power and market position while asking governments to lock in regulations for the field around them.

Over the past fortnight, warnings once treated as fringe doomsaying have moved into everyday conversations, sparking public outcry. (Photo by Karl Mondon / AFP)

For the most worried whistleblowers, however, that is all minutiae.

The frightening world safety campaigners exist in is one where each new problem is treated as an unpredictable development.

Ukrainian AI-governance researcher Anna Mysyshyn was warning US politicians about the reckless implementation of AI in military systems just days before the report on the cargo ship emerged.

“The approval button alone does not demonstrate the control,” she said.

Another source familiar with US military policy was even more direct.

“AI in targeting is definitely something that is ramping up and there is no real guidance for how having a human in the loop will prevent civilian casualties or fratricide,” the source said.

Three Democratic senators — Mark Warner, Jack Reed and Chris Coons — have now demanded an inspector-general investigation into whether the Pentagon and intelligence agencies placed AI “experimentation” ahead of effective governance.

They also want investigators given unrestricted access to other suspected failures in AI-assisted targeting workflows.

Meanwhile, Washington has proposed an AI-incident notification system with Beijing ahead of talks between Donald Trump and Xi Jinping.

Treasury Secretary Scott Bessent said the two powers needed more transparency around failures carrying national-security consequences.

They might just be right.

Home

Devastating AI prediction just came true