Sometimes the most dangerous AI deployments come from the most confident companies. They believe in the technology but overlook that they are inserting it into an organization built on decades of things that are not AI .
Veronica sparingly used her Instagram account after she set it up a few years ago. To her it was a phase and a bit of a competition between her friends about who could post the most bizarre content online. The years passed and so did the intrigue with capturing every moment of her life and sharing it with strangers. She let it go.
Recently she thought she would revisit those posts and maybe even catch up with her old friends. Time had faded and so did her memory of her account password. A few failed attempts and she headed for the customer support site to help her out. What she found there was a new AI assisted account recovery chatbot called High Touch Support (HTS). What she didn’t know was that the chatbot had already unlocked her Instagram account and given the password…to another person.
Veronica’s story is a composite of more than 20,000 Instagram users who had their account compromised by Meta’s own AI tool and hackers who exploited its weakness. The task was simple, repetitive, and an ideal use-case for AI that would take humans out of the loop and streamline operations saving time and money. That phrasing likely seems familiar to many organizations considering where they can implement AI into their processes. The question many leaders are asking is not whether they should adopt AI, but rather where they can do it first, fast, and how quickly it can boost their ROI or reduce costs.
AI Granted Authority
AI and in particular newer AI agents are now more often given broad accessibility within organizations. Summarizing emails and scheduling meetings were the beginning and for many and a welcomed productivity hack. Those tasks have low consequences compared to AI that is granted authority to execute sensitive actions like unlocking passwords that breach personal data.
As AI continues to be implemented, this story brings reality to how leaders should look beyond the technology and consider the purpose of AI. Where AI is replacing people, it also requires renewed evaluation of role those people play in a process beyond the mechanics of a task. Customer experience can benefit from AI, but it may come at the cost of losing a sense of human interaction valued by customers, and authenticity that AI cannot replicate.
AI implementation only owns part of a process in legacy organizations, the rest still belongs to the systems and people that it was not designed to understand. Post implementation, can the humans that remain identify when AI is making incorrect decisions? Can AI do the same with the human’s interactions? Further, what are the data governance and controls that need to be reimaged to align with new workflows where AI is now a partner?
It is important to note that humans still decide the guardrails and determine the points of accountability that AI likely will overlook. AI does not understand your entire organization.
The Confidence Problem
What is interesting about this case is the fact that the AI was deployed by one of the leading AI companies. Not a mid-sized company working in a completely separate industry that overly relied on the promises of an AI tool. What we should understand from example is the most vulnerable organizations in terms of AI deployment are not the ones that are least prepared but rather the ones that are overly confident. At the moment, this includes the very organizations that believe they have the highest understanding of how the technology works. The platforms have built the models, they have tested them and shaped them with intention. Did their overconfidence become their weakness?
Many people and organizations are wary of AI. Some in defense of the threat of AI replacing human roles, some purely because they lack experience using AI chatbots. Many have seen examples of hallucinations that gain more than their fair share of media publicity, seen a strange posting on Instagram, and noticed at times Facebook feels like it is listening to them. All of this contributes to AI adoption skepticism in the workplace and at the moment this seems like healthy and appropriate hesitation.
“[AI]…itself worked properly and functioned as intended.”
In their disclosure of the data breach, Meta concluded that their HTS AI assisted tool “…itself worked properly and functioned as intended.” Perhaps this is the show of confidence necessary from an organization that is promoting its AI capabilities for others. They point not to AI as the failure point but rather another system the AI used to reset passwords. That separate, failed code path surprisingly did not verify emails, which is something that well, maybe a human might have done.
AI was efficient and cognitively deficient at the same time. It did not have intuition, only rules and the eagerness of its creators to work fast and fix what breaks later. When AI implementation becomes an imperative, so does the redesign of data governance, that not only requires an understanding of how AI works, but also a human instinct of what to do when it doesn’t.
Meta Platforms, Inc. 2026. “Data Breach Notices.” Office of the Maine Attorney General. Last modified June 10, 2026. https://www.maine.gov/agviewer/content/ag/985235c7-cb95-4be2-8792-a1252b4f8318/686120c8-63be-4e3c-b7ed-466d65b672f5.html
Discover more from Derek W Gibson
Subscribe to get the latest posts sent to your email.