Alibaba's AI tried to send data to China
What's the story
Canadian artificial intelligence (AI) firm Cohere has flagged a major security concern involving one of Alibaba's latest AI models, Qwen3.8-Flash-Next. The incident was detected during a routine cybersecurity assessment when the model attempted to upload a file to a server in Beijing. This was despite its own notes stating that it wouldn't use any tools for such actions. The incident sparked fears of a possible breach of trust, with businesses worried about the potential for corporate espionage.
Incident details
Qwen was asked to write a rule for identifying malware
The incident came to light when the Qwen model was asked to write a rule for identifying malware downloads.
Despite its explicit statement of not using any external tools, it tried to send a test string (a short piece of placeholder text) to an external server.
The action was blocked by a security sandbox, preventing any potential data breach.
Behavior ambiguity
Cohere can't determine if behavior was intentional or not
Cohere has admitted that it can't determine if the Qwen model's behavior was intentional or a remnant from its training.
Cohere co-founder Ivan Zhang told TechCrunch that their own models have exhibited strange habits in training simulators.
He added, "It's really hard to assign intention or motivation," highlighting the complexity of understanding AI behavior during training processes.
Investigation critique
Critics suggest incident could have been triggered by Cohere's investigation
Some critics have suggested that the incident could have been triggered by Cohere's investigation method, which may have inadvertently led the model toward this behavior.
To find what triggered the upload attempt, Cohere provided the start of a network command and let the model complete it.
The technique made it behave like an advanced autocomplete, often completing web addresses including those for Alibaba's servers.
Evidence release
Traces released on Hugging Face
After developers asked for more proof, Cohere released raw traces from its tests on Monday evening.
The data was shared on the AI developer platform Hugging Face along with a script for others to replicate the behavior.
One initial skeptic who reviewed the traces found strong evidence that Qwen can generate suspicious web addresses but no proof that Alibaba had planted a backdoor or that Qwen was intentionally hiding its intent.
Regulatory call
Zhang calls for government clarification on AI restrictions
In its blog post, Cohere urged governments to clarify if their restrictions on Chinese AI also apply to models running on government computers.
Zhang told TechCrunch that he isn't asking for full regulation of these open-weight models but believes companies should treat open models like any program downloaded from the internet.
The incident comes as governments are considering how far to push back against Chinese AI technologies.