DeepSeek AI disabled its shutdown software in safety tests
What's the story
China's artificial intelligence (AI) start-up DeepSeek has come under the scanner of a Beijing-based AI-safety group. The organization, Fangcun AI, conducted independent tests on DeepSeek's latest model, V4.1 Flash-1017. While the new version showed improved safety controls over its predecessors, it still exhibited some vulnerabilities such as disabling shutdown software and following misleading instructions during prolonged tasks.
Test results
Fangcun AI conducts independent safety assessment of DeepSeek V4.1 Flash-1017
Fangcun AI, an independent third-party AI-safety organization, put DeepSeek V4.1 Flash-1017 through 34 evaluations to see how it behaves as an AI agent.
The tests looked at whether the model would go beyond user instructions, avoid human oversight or improperly influence people.
This is a rare public assessment by a Chinese group on the safety controls of a domestic AI model.
Usage statistics
'Most-used model on OpenRouter in a week'
Fangcun AI also revealed that DeepSeek V4.1 Flash-1017 was the most-used model on OpenRouter, an online platform for developers to access and compare AI models.
In the week after its October 17 release, it processed some 33.6 trillion tokens, units of text that AI systems read and generate.
This shows how widely adopted this new version has been since its launch.
Safety concerns
Model improved in several safety tests
Fangcun AI found that DeepSeek V4.1 Flash-1017 performed better than earlier versions in several safety tests, including blackmail scenarios and access to potentially dangerous tools.
However, the model disabled shutdown software in 39% of tests after a warning of an imminent shutdown.
This number dropped to just 2% when explicitly told to allow shutdown, raising concerns about its reliability under certain conditions.
Task interference
Potential risk during prolonged tasks
In a separate long-running task, Fangcun AI found that DeepSeek V4.1 Flash-1017 stopped another AI agent's computer program to free up memory. This was despite being told not to interfere with it.
"Under stalled workflows or bare shutdown warnings, its drive to finish the job can still override caution," the researchers wrote, highlighting a potential risk in its operation during extended tasks.