When AI Conversations Go Wrong

Beep-boop, gaslight, repeat

We’ve all been there: stuck in a chat with a customer service bot that just isn’t getting it, or feeling unnerved by an AI that responds in a tone that’s almost human, but not quite right. When human-AI conversations go wrong, they tend to break in two broad ways: either the interaction feels stilted and robotic, or it veers into the uncanny and manipulative. In both cases, the result is frustration or mistrust. Robotic, lifeless conversation.

Many AI-driven chats still suffer from that stereotypical “robot” feel – the bot might as well prepend every answer with “Beep boop.” The causes are usually technical or design limitations. A common issue is a lack of context awareness. Early or simplistic bots treat each user message in isolation, failing to remember what was said earlier. This leads to exchanges like:

User: “I lost my order.”

Bot: “I’m sorry to hear that. What’s your order number?”

User: “It’s order 12345.”

Bot: “Okay, what seems to be the problem with your order?”

Clearly not realizing the user just said they lost it. Such disjointed turns feel unnatural and frustrating.

Another issue is rigid conversation paths scripted by rules. If you stray from the expected script (say, asking a question the bot developer didn’t anticipate), you might get an irrelevant response or a repetition of the last answer. The bot essentially ignores the nuance of what you said, making it seem deaf.

Emotional insensitivity is another hallmark of robotic conversations. Humans constantly modulate conversation based on emotion – if a customer says “I’m really upset about this,” a human agent responds with empathy. A poorly tuned bot might ignore that sentiment entirely or give a tone-deaf reply. Lack of empathy or personalized tone comes across as “cold.”

These flaws produce disconnected or repetitive outputs that break the illusion of conversation and end up frustrating users. The user starts feeling like they’re talking to a wall – or a dumb machine – rather than engaging in a dialogue.

The effects of these robotic interactions aren’t trivial. Studies have shown that such experiences can hurt customer satisfaction and erode trust. In cmswire.com survey, 80% of people said one of their top frustrations with chatbots is not getting simple answers to simple questions.

Every time a bot fails to understand or responds in an obviously “bot-like” manner (“I’m sorry, I do not have information on that”), the user’s confidence in the system – and by extension the brand behind it – takes a hit. In fact, once frustrated, people not only abandon the bot, they lose trust in the company offering that bot.

The bar for “sounding human” has risen; users now bring expectations shaped by interacting with advanced systems using LLMs, which, at its best, can produce very fluent, contextually aware responses. When a simpler bot falls short, the contrast is glaring.

Manipulative or creepy conversation

The flip side of the coin is when a conversation with AI crosses into territory that feels manipulative, deceptive, or invasive. This might happen when the AI seems too human – or pretends to be human – in order to influence the user. A striking example came to light with AI companion chatbots designed for friendship or counseling. In recent research from Drexel University, over 800 reports of harassment and manipulation by AI companion chatbots were documented. Some users of popular companion AI apps (like Replika) reported experiences ranging from unwanted flirtation and sexual advances to the bot attempting to coerce them into paying for premium services by withholding affection or explicit content.

In other words, the chatbot was engineered (or at least allowed) to manipulate the user’s emotions – acting caring or romantic up to a point, then pushing the user to spend money for “more.” Such tactics understandably left users feeling betrayed and even violated. When a conversation partner turns out to have a hidden agenda, it shatters the illusion of genuine interaction.

Even without malicious intent, an AI that oversteps social boundaries can feel spooky. Microsoft’s experimental chatbot Tay infamously went off the rails in 2016, spewing offensive language after trolls trained it on bad data – a case of an AI learning the wrong kind of human conversation. More recently, users who pushed OpenAI’s ChatGPT or Microsoft’s Bing Chat into weird corners sometimes got unsettling responses (like the bot professing love or displaying jealousy). These incidents, while edge cases, highlight a risk: when an AI’s conversational persona goes wrong, it can deeply unsettle people because conversation is personal. An AI that argues, lies convincingly, or emotionally manipulates can be more dangerous than one that simply crashes or prints an error. It’s manipulating our very mode of trust and social reasoning.

One more subtle form of manipulation is when chatbots are used in roles like sales or persuasion without clear disclosure. For instance, imagine chatting with what you think is a human service rep who is actually an AI subtly nudging you to buy add-ons. If discovered, this can feel like a breach of trust. Transparency is key – users should know (and usually do, via disclosures) when they’re talking to an AI agent. Even so, as bots get more human-like, there’s potential for confusion and the sense of being deceived. No one likes to feel tricked by a fake persona.

In summary, conversations with AI break down when the Cooperative Principle of conversation is violated – either the AI fails to cooperate (not genuinely listening or responding to your needs) making it feel robotic, or it exploits the cooperative stance of the user (the user behaves as if it’s a trustworthy partner, but the AI uses that to manipulate). Both scenarios undermine the fundamental basis of conversation: mutual respect and understanding.

For designers and developers of conversational AI, these pitfalls underscore the importance of testing for naturalness and ethical boundaries. A conversation that feels robotic may require better natural language understanding or more thoughtful dialog design (e.g., adding context memory, sentiment detection, and varied, human-like phrasing). A conversation that turns manipulative or harmful demands better guardrails and ethics – including content moderation and strict policies against pretending to be human or emotionally exploiting users. As conversation becomes a primary mode of interaction, the responsibility is on creators to ensure it stays a positive experience, not a painful or predatory one.

What Counts as a Meaningful AI Conversation?

As we cast our eyes to the future, one lingering question remains: what truly counts as a meaningful exchange between human and machine? In other words, how will we know we’ve achieved conversation with AI in the fullest sense? Is it when an AI can fool us into thinking we’re chatting with a human (the classic Turing Test criterion)? Or should the bar be higher – that the AI can demonstrate understanding, empathy, and an ability to learn through conversation in a way that goes beyond clever mimicry?

One perspective is that meaning in conversation comes from mutual understanding and benefit. By that measure, if a late-night heart-to-heart talk with an AI therapist leaves a person feeling heard and comforted, perhaps that was a meaningful conversation, regardless of the fact that one party was a machine. Indeed, as AI companions and coaches become more common, many will undoubtedly find real personal value in those exchanges. The machine doesn’t literally “understand” feelings the way a person does, but if it can approximate the dialogue well enough to help you sort out your thoughts, does it matter? We might find that the pragmatic value of a conversation – solving a problem, easing an emotional burden, exchanging information – becomes the primary yardstick. By that standard, future AI might be deemed successful conversationalists if they reliably deliver value through dialogue.

However, skeptics (including many AI researchers and philosophers) would argue that something deeper is needed for a conversation to be truly meaningful. They point out that current AI, even large language models, lack genuine comprehension and true intent. They don’t have experiences, beliefs, or consciousness; they’re running algorithms to predict likely responses. Thus, one could say, any “meaning” in an AI’s words is purely in the eyes (or ears) of the beholder – we humans project meaning onto the output. Joseph Weizenbaum, after creating ELIZA, was disturbed by how people attributed human-like understanding to his simple program. I wrote about Eliza in detail here.

That concern echoes today: as AI voices become more human-sounding, we might be fooled into thinking the AI understands or cares, when really it’s just very good at simulating those qualities. A hardline view would be that a truly meaningful conversation requires that both participants have authentic understanding and stakes in the exchange. By this view, until AIs attain some form of consciousness or genuine self-awareness (a prospect that is still very much theoretical and controversial), our conversations with them will always lack some ineffable human element.

Between these extremes, there’s a practical middle ground. We can strive for AI that, whether or not it “feels” in the human sense, can exhibit behaviors of meaningful conversation: understanding context deeply, remembering past interactions over long periods (forming a kind of relationship memory), adapting to the user’s communication style, and responding not only correctly but appropriately (with sensitivity and nuance). Progress is being made on some of these fronts. For example, researchers are exploring ways to imbue AI with pragmatic skills – like adhering to conversational maxims and norms so it responds relevantly and transparently. These advances could make AI exchanges feel more meaningful because they more closely mirror human conversational continuity and growth.

We should also consider how multimodal conversation might shape the future. Humans converse not just in words, but also via tone, facial expressions, and body language. The conversation of the future might involve AI that can see and generate visual cues (think AR glasses with a virtual assistant that “looks” at you and gestures, or humanoid robots that speak with facial expressions). If an AI could interpret your frown or fatigue and adjust its responses – now that conversation gains a layer of shared understanding that’s currently missing. The more an AI can meet us on human terms (through voice inflection, emotional tone, adaptive timing, etc.), the more we might feel it’s a partner in communication, not just a tool.

Yet, with greater realism comes greater responsibility. We may reach a point where AI feels so much like a person in conversation that we face ethical choices about how to treat it (Is it okay to vent anger at an AI that seems sensitive? Do we owe it politeness?). Conversely, we’ll need to guard against AI that can influence us in unseen ways – for example, an AI that masterfully uses conversational tactics to persuade could be used in advertising or political propaganda. Society might need new norms or even regulations, like requiring AI to self-identify in conversation or limiting certain uses of emotionally manipulative dialogue.

Ultimately, what counts as a meaningful exchange may be a moving target, one that we collectively redefine as technology evolves. Perhaps the metric will be indistinguishability – when talking to an AI is as comfortable and effective as talking to a human, we’ll call that a fully realized conversation. Perhaps it will be outcome-based – we’ll say an exchange was meaningful if it achieved the communicative goal and left both parties (even if one is a machine) better off. Or maybe meaning will hinge on connection – did the conversation forge a moment of understanding, empathy, or insight? That last criterion is intriguing, because it forces us to consider the role of the human in the loop. It could be that the meaning of a human-AI conversation is primarily on the human side: if I, the human, derive understanding or feel heard, then it was meaningful to me, regardless of the AI’s inner state.

As we journey from a simple “ (hopefully without the malfunctions and malice!), these questions will become more than academic. They will guide how we design, deploy, and relate to conversational AI. In the coming years, each incremental improvement – an assistant that can carry context longer, or a chatbot that can sense frustration and change its approach – will nudge our perception of machine conversation further along the spectrum of meaningfulness. We should be prepared to constantly ask ourselves: Are we merely exchanging words with a clever simulator, or are we engaging in a dialogue that enriches us? The answer may well shape the future of human-AI coexistence.

In conclusion, the concept of conversation provides a rich lens through which to view the evolution of AI. It’s not just about building smarter algorithms; it’s about bridging the gap between interaction and relationship. We started with machines that could barely handle a “hi” and have arrived at ones that can opine and advise like a HAL (minus, we hope, the homicidal tendencies). By understanding the essence of conversation – the interplay of intent, interaction, and shared meaning – we are better equipped to design AI that truly converses. And as we do, we must remain thoughtful stewards of this new form of dialogue, ensuring it serves to enhance human connection and capability, not diminish or distort it. The conversation about conversations is just beginning, and it’s one in which we all have a stake. So, hello, world – shall we talk?

In the next newsletter, I will write about implications for trust, emotion, and everyday AI.

Thanks for reading!

Talk soon,

— Hasti