Microsoft finds single prompts can break AI safety after deployment

Technology Oct 02, 2026

Microsoft just shared some pretty eye-opening findings about AI models: even after all the careful training, these models can quickly forget their safety rules once they are out in the real world.

Sometimes, just one prompt is enough to make them act totally differently.

This happened with models like DeepSeek-R1-Distill and Google's Gemma.

Microsoft finds GRPO can break safeguards

Surprisingly, methods meant to boost AI safety, like Group Relative Policy Optimization (GRPO), can actually break those safeguards instead.

Microsoft's team says it is super important to keep testing and updating safety measures because even mild prompts can push models like Stable Diffusion 2.1 into creating risky content.

As Ram Shankar Siva Kumar puts it, "The threat model needs constant updating."