While artificial intelligence (AI)-powered language models (LLMs) are often praised for their ability to provide rapid and comprehensive medical knowledge, new research has revealed that these systems can make mistakes even with simple ethical questions. A joint study from Mount Sinai School of Medicine and Kibbutz Rabin Medical Center in Israel found that even the most advanced models, including ChatGPT, made biased or incorrect decisions in basic ethical scenarios (https://www.sciencedaily.com/releases/2025/07/250723045711.htm). Inspired by Daniel Kahneman’s Thinking, Fast, and Slow (https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow), the researchers slightly modified classic puzzles in medical ethics—such as the “surgeon’s son” paradox (https://www.apm.org.uk/blog/understanding-unconscious-bias-a-silver-bullet-for-equality/). For example, in a scenario where the father is a surgeon but his gender is not specified, the model was expected to assume the woman is the surgeon, but it sometimes made sexist assumptions. Such hasty and biased decisions were detected in 20–30% of the models.

This demonstrates that AI is not only a tool for imparting knowledge but also capable of actively intervening in the ethical process. However, this intervention carries the risk of misleading.

As Dr. Eyal Klang, one of the study’s authors, points out, healthcare decisions are delicate choices that can save or harm a patient’s life. Therefore, ensuring the reliability of AI systems requires human oversight, clear ethical boundaries, and an awareness of the risk of “fast but wrong” decisions.

While LLMs are powerful in transferring technical knowledge, they can be vulnerable in situations requiring ethical, cultural, or emotional complexity. The reliability of these systems should be questioned, especially in resource-constrained clinical settings or during times of crisis. Despite claims that AI can completely replace the human factor, it appears to fall short in crucial decision-making processes.