How AI error nearly triggered US-China military confrontation
What's the story
The US military narrowly averted a potential international crisis this spring when an artificial intelligence (AI)-generated intelligence report falsely claimed that a Chinese ship in the Middle East was carrying nuclear weapons program components. Citing four sources familiar with the incident, CNN reported that armed members of the US military were preparing to board the vessel and military planes were in the air.
Misidentification
'Report entirely false'
The false intelligence report was prepared by an analyst from the special operations command, who had used an AI chatbot for assistance.
The tool wrongly identified the cargo on the Chinese ship, leading to a hasty response from US military officials.
One source told CNN that the AI report was "entirely false," and it "almost started a war," highlighting how any action against a Chinese vessel could have escalated into armed conflict between the two nations.
Adoption concerns
Incident highlights dangers of AI in military operations
The US military and intelligence community are increasingly integrating AI into their operations, from analyzing raw intelligence data to managing logistics and supply chains.
However, this incident highlights the potential dangers of using such a powerful but poorly understood technology for targeting during wartime.
Analysts have long warned that AI could result in catastrophic miscalculations if nation-states rely on inaccurate or corrupted data.
Chatbot involvement
Analyst used chatbot to identify ship's cargo
In this case, the analyst asked a chatbot about some intelligence reporting on the ship's manifest from US Special Operations Command Pacific in Hawaii.
It is unclear if the chatbot was a commercially available one or a US government product.
The bot combined open-source intelligence with secret signals intelligence in government holdings and concluded that the Chinese ship in the Middle East was transporting components of a nuclear weapon.
Decision-making
US military increasingly using AI for battlefield decisions
The rapid adoption of AI is aimed at helping the military make faster battlefield decisions, such as which targets to strike or where to move military assets.
However, this effort is decentralized, with different parts of the government using different tools under different orders and safety standards.
The wide variety of AI systems being used by various sectors of the military and intelligence community means that their reliability and functionality vary widely.