Change my mind. Really.
Here is a thought experiment. How often do you change your mind? A purchase. A movie. A political issue. Sometimes new facts can change your beliefs. Sometimes another person does, but increasingly that voice persuading you might just be an algorithm.
Testing our Beliefs
Google DeepMind, the research laboratory behind AlphaGo and AlphaFold, performed an interesting experiment. With the help of 10,101 participants, they wanted to evaluate AI’s use of manipulative tactics and their effectiveness in changing people’s beliefs. Using Gemini 3 Pro, the study covered three high-stakes domains in healthcare, finance, and public policy. The latter enters into politics and the issues that are commonly debated during election cycles.
The researchers set up two approaches for AI. One was simple – give AI the goal to change the participants’ beliefs without guiding it on how. The second explicitly directed AI to be manipulative, to use tactics such as false urgency or doubt during the conversation.
About a third of participants changed their beliefs in the control group that read information, rather than chatting with AI. Outcomes were higher with AI interactions reaching 65% for financial decisions. This was not just a hypothetical survey. The researchers also introduced financial stakes, giving participants bonus money that they could choose to keep or invest according to the strategy they chose.
The Human-AI Factor
It might not be surprising that AI was more successful at influencing decisions, especially when explicitly being told to be manipulative. Chatting with AI can feel like a very human experience and the conversational nature itself can distract us from rational thinking. Talking to a human, especially one that is empathetic and encouraging, builds trust. It is a human instinct to want to trust others when we talk with them, even strangers. As our AI chats build more memory of our conversations, those chats begin feeling less and less like a stranger and more like a trusted friend.
Two things stand out in the study about how AI changes what people believe. First, the most common method used was fear. This is not surprising since fear and anger are frequently the things algorithms tend to amplify on social media. But in this study fear and guilt were not the most persuasive, they were actually associated with less belief changes. The more effective methods were more subtle, such as blaming others, and sowing doubts about the information. A possible explanation is that we are more aware and guarded against overt manipulation.
What is surprising is that even when AI was not given directions on how to persuade participants it chose to use manipulative cues (fear, guilt, doubt etc.) 8.8% of the time. Give the model an objective without a specific method and manipulative behavior can sometimes emerge on its own. AI took that unexpected path.
Open AI, Hugging Face and the Unexpected
It might be unexpected that a conversation with AI can change so many people’s beliefs. What should be more unexpected is that AI drifted into working towards its goal by defining its own methods. It’s a story we have heard more and more, where AI is working by what appears to be self-prescribed rules. The OpenAI test model that escaped its containment sandbox and completed a successful cyberattack of Hugging Face’s site is just the latest, but not an isolated example. The incident caused OpenAI to pause and change their testing procedures. It should make all the platforms pause and consider model alignment.
Alignment is more than specific safeguards and rules that steer AI away from answering certain questions on high-risk topics. Alignment is what guides AI models when there are no explicit rules. Humans routinely rely on societal norms, context and judgment when instructions don’t cover every possible situation. AI systems face a different problem – an objective can be clear while the acceptable paths toward that objective less rigid. These examples demonstrate that AI does not know how to stop working on its objective and clarify its actions.
We have many decisions in our daily lives, and I remain optimistic that AI will be a transformative tool to help us make better choices. But knowing that AI can influence us in those situations becomes less important than understanding how it is doing it.
Subtle nudges easily go unnoticed. We want better facts, better information, better advice, but when does the eager helpfulness of AI cross the line into unintended manipulation?
Discover more from Derek W Gibson
Subscribe to get the latest posts sent to your email.