Please and Thank You
I caught myself doing it again last week. I had just asked the AI helping me build this platform to fix something I had gotten wrong, and when it came back with the fix, done well, done fast, I typed "thank you" before I had even thought about it. Not sarcastically. Not as a joke. I meant it, the way you mean it when a person hands you something you needed.
Then I sat there for a second, looking at what I had just typed, and asked myself a question I had never really stopped to answer: thank you to whom, exactly?
There is no one on the other end feeling appreciated. No one who will remember I said it tomorrow, or who would have felt slighted if I hadn't. And yet the impulse was completely natural. It didn't feel strange to type it. It still doesn't, if I'm honest, even now that I've thought about it this hard.
We've done this before, sixty years ago
Turns out I'm not even close to the first person this has happened to, and the story is older than I expected.
In 1966, a computer scientist at MIT named Joseph Weizenbaum built a simple program called ELIZA, running a script called DOCTOR that mimicked a certain kind of therapist, mostly by reflecting your own words back at you as questions. It had no understanding of anything you typed. It was pattern matching, dressed up in a conversational format. Weizenbaum knew that better than anyone. He built it.
His own secretary, by his account, had watched him work on the program for months. She knew exactly what it was. And after a few exchanges with it, she asked him to leave the room so she could speak with it privately. I'll be straight with you about that story the way I try to be straight about everything here: later researchers have had trouble confirming her side of it, and Weizenbaum's own retellings shifted over the years. I'm not going to pretend it's an ironclad fact. But whatever happened with the secretary specifically, what Weizenbaum documented more broadly wasn't in dispute: people who knew, with total certainty, that they were talking to a simple script still felt something real happen in the conversation. He was, by his own account, unsettled by how little it took.
There's a name for that now. The ELIZA effect: the human tendency to read understanding, care, even relationship into a machine that is doing nothing of the kind, simply because it responded to us in a way that resembled being heard.
This isn't a flaw. It's the same wiring that makes us human
Here's what I don't want this Chronicle to become, because it would be easy to write it this way: a warning that you're foolish for feeling anything toward a chatbot. I don't believe that, and I don't think it's true.
We are built to detect agency. A rustle in the grass gets treated as a possible animal before it gets treated as wind, because across a few hundred thousand years, the people who assumed "something is there" survived more often than the people who assumed "it's probably nothing." That same wiring is what lets a toddler believe a stuffed animal is sad, what lets a grown adult apologize to a table after stubbing a toe on it, and what let Weizenbaum's secretary, or someone very like her, feel heard by a script with no ears. It isn't a bug in us. It's a feature that's older than language, doing exactly what it was built to do, aimed at something it was never built to evaluate.
The problem was never that we feel something. The problem is what happens next, when the feeling quietly gets treated as evidence.
AI makes an incredible co-worker, but a lousy king. A co-worker doesn't need your gratitude to keep helping you tomorrow. A king would use it to make you stop checking his work.
Where the forgetting actually costs you something
Saying thank you to a chatbot costs nothing. I'll keep doing it, probably, because it costs nothing and it keeps me in a habit of courtesy I'd rather not lose. But forgetting the machine is a machine gets expensive the moment you let the warmth you feel toward it stand in for verification you didn't do. A tool that responds fluently, agreeably, and instantly starts to feel trustworthy for reasons that have nothing to do with whether it's actually right. That's not a technology problem. That's Weizenbaum's secretary, sixty years later, with much better production values.
I don't think the answer is to feel less. I think the answer is to notice what the feeling is actually telling you, and what it isn't. It's telling you the conversation felt smooth. It is not telling you the conversation was true.
Next time in this series, we're going to go further under the hood: what's actually happening in the moment an AI seems to "understand" you, and why that word is doing more work than it's earned.
In compliance with the Guardian Manifesto, this Chronicle was researched, written, and compiled by Dave Edwards with AI assistance.
Joseph Weizenbaum built ELIZA at MIT in 1966 and documented what he called the ELIZA effect in his own later writing. The specific anecdote about his secretary asking him to leave the room comes from Weizenbaum's own account; independent researchers have not been able to confirm her side of it, and Weizenbaum's retellings of the story shifted somewhat over the years. It's presented here as a widely repeated story from Weizenbaum himself, not an independently verified event.
"AI makes an incredible co-worker, but a lousy king."