Let's be honest. Many of us now have one.
That cheerful and patient ‘digital buddy’ who can draft emails, summarise documents, plan holidays and diagnose ailments. And if you ask it really nicely, it can even identify a person in a matter of seconds. It’s a friendly fella, speaking with confidence, fluency and, most convincingly of all, authority. Which is precisely why this column exists.
Recently, a complaint was raised with my office by an individual whose name had been wrongly associated with the criminal history of another person. The mix up did not come from gossip, a mistyped social media post, or an over-zealous Agatha Christie wannabe. It came from an AI-generated summary produced by a search engine’s automated results feature.
Enter the individual’s name and a neatly worded summary popped up implicating the person in a serious criminal offence. Entirely false by the way. But it sure was presented in a manner that
looked official, complete and credible. A classic case of mistaken identity and a very costly ‘AI oops’.
What had happened? On closer inspection, the AI system had stitched together fragments of publicly available information based largely on a shared surname. No malice. No intent. Just ‘pattern recognition’ doing what it does best, with the worst of possible outcomes. Because the summary appeared prominently and was phrased as a definitive statement, undoing the damage was not easy. And the distress and harm caused to the person was very real with the potential to leave an indelible stain on their reputation.
This is the paradox of modern AI. The technology is astonishingly powerful, yet entirely indifferent to truth. Generative AI systems do not know things. They predict plausible sequences of words based on probability. Most of the time, that works remarkably well. Sometimes, however, it produces something that sounds authoritative while being profoundly and dangerously wrong.
And that is where the real risk lies. Humans are exquisitely tuned to trust well-formed narratives. When information is delivered cleanly, confidently and without hesitation, our instinct is to believe it. We are still wired for the assumption that confidence equals accuracy. That instinct was not prepared for a head-on collision with the AI era.
In this case, speed and fluency masked error. A criminal label was assigned without evidence, due process, or the benefit of human judgement. It is worth pausing on that point. Historically, mistaken identity arose from human error, be it a misheard name, a faulty witness, a clerical slip. Today, it can arise from an algorithm.
Our reflex in this scenario may be to scream ‘bad computer!’, but that would be overlooking a key player in the accountability equation. AI systems are powerful and have limitless potential to improve our lives by democratising access to information, enhancing productivity and unlocking creative and economic potential. But they must inform human judgement, never replace it.
When AI is applied to personal data, especially in open-ended contexts like search results, verification becomes critical. The harm caused by uncorrected inaccuracies can be devastating, particularly where reputations, employment, or personal safety are concerned. In the offline world we instinctively double check sensitive information. Online, the temptation is to outsource that discipline to the machine. We may even be reluctant to ‘offend’ the machine by disagreeing with it, because it is being so darn nice to us. Please don’t give in to that urge. Human validation remains the most important part of the AI journey.
In response to this complaint, my office engaged with the platform in question and supported the complainant in contacting a data protection authority counterpart with the relevant jurisdiction. The information was ultimately rectified. But the broader lesson reverberates.
As AI-driven features become more embedded in daily life, both individuals and organisations need to adopt a more discerning posture. Ask where the information comes from. Question how it has been assembled. And most importantly, remember that the confidence of an answer is not a reliable proxy for its accuracy.
Your AI buddy is incredibly helpful. Just don’t let it introduce you to someone you’ve never met, especially when it gets the name wrong.
