24/09/2025
From Mark Jennings-Bates' latest piece on the truth about artificial intelligence:
" To understand why AI systems are opaque, we need to examine how modern machine learning actually works. Unlike traditional software, which follows explicit rules programmed by humans, AI systems learn patterns from data through processes that are fundamentally statistical rather than logical.
Consider a deep neural network – the architecture underlying most contemporary AI applications. These systems contain millions or billions of parameters, each representing mathematical relationships learned from training data. When data is put into the system, it flows through multiple layers of interconnected nodes, with each layer performing complex mathematical transformations.
The problem is that these transformations, while mathematically precise, do not correspond to concepts that are understandable to humans. A loan-approval AI might base its decision on thousands of subtle correlations between applicant characteristics, but these correlations may not translate into explanations that humans can comprehend or validate.
Recent research from the Massachusetts Institute of Technology illustrates this challenge starkly. Researchers studied a state-of-the-art AI system used for medical image analysis and found that even when the system made correct diagnoses, the reasoning process involved interactions between millions of parameters in ways that defied human interpretation.
The complexity is not accidental – it is often the source of AI’s power. The ability to identify non-obvious patterns and subtle correlations that humans might miss is precisely what makes AI valuable. But this same capability makes AI decisions inherently difficult to explain in terms that humans can understand and validate. "
In 2019, a major healthcare system implemented an AI tool to help doctors identify patients at risk of sepsis – a condition that kills more than 250,000 Americans annually. The system analyzed patient data and flagged high-risk cases with impressive accuracy during testing. But when doctors tried ...
