Microsoft's new AI beats doctors at diagnosing complex health conditions
What's the story
Microsoft has unveiled a new artificial intelligence (AI) system, capable of outperforming human doctors in diagnosing complex health conditions. The innovative technology was developed by the company's AI unit, led by British tech pioneer Mustafa Suleyman. The system mimics a panel of expert physicians when tackling "diagnostically complex and intellectually demanding" cases, Microsoft said in a blog post announcing the research.
Performance comparison
AI system solves 80% of diagnostic challenges
When paired with OpenAI's advanced o3 AI model, Microsoft's system "solved" over eight out of 10 case studies specifically designed for the diagnostic challenge. In contrast, practicing physicians without access to colleagues, textbooks or chatbots only had an accuracy rate of two out of 10. This stark difference highlights the potential of AI in complex health diagnoses.
Cost efficiency
More efficient at ordering tests, claims Microsoft
Along with its diagnostic prowess, Microsoft's AI system is also more cost-effective than human doctors. The tech giant says the system is more efficient at ordering tests, thus saving on healthcare costs. Microsoft has also addressed concerns about job losses in the medical field, emphasizing that it sees AI as a tool to assist doctors rather than replace them.
Research rationale
AI passing US Medical Licensing Examination is not enough: Microsoft
Microsoft also questions the effectiveness of AI in passing the US Medical Licensing Examination, a crucial test for medical licensing in the US. The company argues that these multiple-choice tests emphasize memorization over understanding, which could "overstate" an AI model's competence. To address this issue, Microsoft is developing a system that mimics a real-world clinician's step-by-step approach to diagnosis.
Testing methodology
How the researchers tested the AI models
The new Microsoft approach employs complex case studies from the New England Journal of Medicine (NEJM). Suleyman's team turned over 300 such studies into "interactive case challenges" to test their method. The company used existing AI models, including those from OpenAI, Meta, Anthropic, Grok, and Google's Gemini. It then employed a bespoke agent-like AI system called a "diagnostic orchestrator" to work with each model on what tests to order and what the diagnosis might be.
Future implications
Tech not ready for clinical use yet
Microsoft's AI system showed a "breadth and depth of expertise" that went beyond individual physicians because it could span multiple medical disciplines. The company said, "Scaling this level of reasoning - and beyond - has the potential to reshape healthcare." However, Microsoft also acknowledged that its work is not yet ready for clinical use and further testing is needed on its "orchestrator" to assess performance on more common symptoms.