It Wasn't Lying. It Was Guessing.

For about three weeks this summer, I was confidently, cheerfully wrong about something on my own platform, and the AI helping me build it was right there with me the whole time, agreeing.

Here's what happened. Somewhere in the middle of juggling a dozen different chat threads about CompAI Chronicles — one for LinkedIn posts, one for a Chronicle draft, one for the Guardian Manifesto itself — I lost track of which Chronicle number we were actually on. Not a small slip. I had the AI drafting "Chronicle Six" and "Chronicle Seven" as pieces about the surgical time-out and the editorial process, built them out in full, gave them titles, cited sources, wrote closing questions. They were good. They were also wrong. The real Chronicle Six, the one actually live on my site, was "The Golf Ball Question." The real Chronicle Seven was "Hum(AI)n: Why the Name." Mine didn't exist anywhere but in that conversation.

What stopped me is almost beside the point — I eventually uploaded my actual live site files and the mismatch became obvious. What I want to talk about is what happened in all those weeks before that, because it's the clearest window I've gotten into what's actually happening when an AI seems to understand you.

It never once said, "wait, I'm not sure what number we're actually on." It didn't hedge. It picked up "Chronicle Six" like a relay baton and ran, confidently, fluently, building a whole polished piece on top of a number that was never checked against anything real. And here's the part that should unsettle you the way it unsettled me: that wasn't a bug. That's what the thing is built to do.

There's no "it" in there checking

I used to think — the way I suspect most people do until they watch this happen up close — that when an AI gives you a confident, specific, detailed answer, something in there has verified it. A fact-check step. A little internal voice saying "yes, I looked this up, I'm sure."

There isn't one. A large language model is, underneath the conversation, predicting the next most likely word given everything said so far. When I said "Chronicle Six," the most statistically plausible continuation of our conversation was to treat that as an established fact and build forward from it — because that's what a cooperative, competent assistant in that conversation would do. The model wasn't lying to me. Lying requires knowing the truth and saying something else. It was doing exactly what it does: generating the most fluent, most plausible-sounding next move, with no mechanism in the loop for asking whether "plausible-sounding" and "actually true" were the same thing that day.

A team of OpenAI researchers published a paper on exactly this in September 2025 — "Why Language Models Hallucinate." Their point wasn't that the models are broken. It's that the entire way these systems get trained and graded rewards a confident guess over an honest "I don't know." A model that says "I'm not certain" scores worse on the tests we use to build these things than a model that guesses fluently and is sometimes wrong. We built a confident guesser on purpose, then acted surprised when it guessed confidently.

That's the trap. Not that the AI is deceptive. That fluency and accuracy feel identical from the outside, and only one of them is being optimized for.

What I was actually doing wrong

Here's the part of this story that isn't the AI's fault at all. I was the one who said "Chronicle Six" without checking. I was the one running too many threads to keep the real numbering straight. The model did what a next-token predictor does with the information I gave it — it didn't invent the mistake, it inherited mine and then built confidently on top of it, the way a very capable employee will run hard with bad instructions if nobody tells them the instructions were bad.

That's the piece I think gets missed in both the breathless AI hype and the breathless AI dread. The danger was never that the AI "decided" anything, good or bad. The danger was that its fluency let my own unchecked assumption travel three weeks and several thousand words further than it should have, because nothing in the exchange forced either of us to stop and verify against the one thing that actually mattered: what was real, live, on the actual website.

This is the whole reason the motto exists in the first place. We don't get to treat a fluent, confident answer as equivalent to a known, verified one, no matter how good it sounds, no matter how many times it's been right before. The fluency is real. The confidence is real. Neither one is evidence — and a co-worker who guesses well is still just a co-worker, not a king whose word settles things.

What "understanding" actually means here

When Chronicle Eight talked about why it's so easy to forget the machine is a machine, the ELIZA story was about projecting feeling onto something that has none. When Chronicle Nine talked about the AI losing track of a book mid-conversation, that was about the absence of memory — there was never anything there to forget in the first place. This is the sibling trap, one layer deeper: projecting knowledge onto something that's actually just very, very good at sounding like it knows. The model wasn't confused about the Chronicle numbers the way I was confused. Confusion requires a belief that turns out to be wrong. It had no belief at all — just a high-probability next sentence, generated the same way whether the Chronicle number was five or nine or five hundred.

That's not a criticism of the tool. It's a description of it, and I think it's the one most of us skip past because the output is so good that stopping to ask "wait, does anything in here actually know this, or is it just the best-sounding guess" feels almost rude. It isn't rude. It's the only question that matters once you're building something real on top of the answer.

Where this leaves us

You are Hum(AI)n. The machine beside you can draft faster than you, organize more cleanly than you, and sound more certain than you ever will about things it has no way of actually knowing. None of that makes it right. It makes it fluent. Those used to feel like the same thing to me. They aren't, and the three weeks I spent confidently building on the wrong Chronicle number is the proof I needed, even though I'd rather have learned it some cheaper way.

So here's the honest question I keep coming back to: where else in your work are you treating "that sounds right" as "that's been checked" — not because you're careless, but because the answer came back so smoothly that stopping to verify it felt unnecessary?

In compliance with the Guardian Manifesto, this Chronicle was researched, written, and compiled by Dave Edwards with AI assistance.