The failures my metrics hid

The guide on this site answered every question it was asked for three days running. Perfect coverage, by the only number I had. It was also, twice inside one minute, confidently wrong about itself, and the visitor left. This is what a coverage metric structurally cannot see, what it cost, and the tests I wrote so that this class of failure cannot hide again.

A visitor asked my chatbot whether AI was involved in the conversation. It answered by telling them about the AI agents I design. They asked again, more precisely. It said the same thing.

The number agreed with me

Every turn in this chat is logged with one flag: whether an answer was found. Divide the answered turns by the asked turns and you get coverage. On 12 August the log said one hundred percent, and it kept saying so.

There are two ways this bot can fail, and the flag only knows about one of them.

A miss is a turn where nothing matched well enough. It is recorded as a miss. It costs the visitor a few seconds, because the bot says plainly that the question is outside what it holds and hands over my email address. The failure is visible to them and to me, and it is cheap.

A confident wrong answer is a turn where something matched and the wrong thing was said. It is recorded as a success. The flag cannot tell the difference, because from inside the engine there is no difference: an answer was found.

So the metric I had been reading every day was incapable of showing me my worst failures, and it was quietly encouraging about them. I built the logging and I chose that flag, so this is not a story about a dashboard somebody sold me. It is a story about measuring the thing that was easy to measure and then trusting it.

Sixty-seven seconds

Here is the session, as the log has it.

Twelve seconds before the first wrong answer, this person had asked what the bot was and received the right reply, which says that every answer here is written in advance and nothing is generated. Then they asked the obvious follow-up, and the engine did exactly what I built it to do. The word ai is a strong signal for the card about my conversation design work. Nothing in the system had claimed the question is AI involved in this conversation, so the highest score won, and the highest score was about me.

On the page whose entire argument is that nothing here is generated, the question is AI involved? got the wrong answer. Twice. The second time it was asked with the ambiguity removed.

The only turn the log flagged in that session was this is weird, which is the reaction. The cause was counted as two successes. And this is weird got the generic repair line, which reads a comment about the experience as a failed question, so the last thing that visitor saw was the bot misunderstanding their complaint about being misunderstood.

A promise I never measured

Two days later a reader told me that a sentence in the bot was not accurate. The sentence was this: You can write to me in Romanian. I understand it, and I answer in English.

I had written that in good faith. I had never tested it. So I wrote forty-eight Romanian questions the way a visitor types them, which is not the way my patterns are written, and ran them through the router. Seventy-three percent reached an answer. The things that fell through were not exotic. Care e emailul ei, what is her email. Cine te-a facut, who made you. Nu inteleg, I do not understand.

Underneath sat a specific and rather beautiful bug. In Romanian the language is română and the country is România, and once you strip the diacritics for matching, those two words are one letter apart. My matcher forgives one typo in words that long, on purpose, because visitors misspell things. So every request to switch language scored on the card about where I live. Two of the three Romanian speakers in the log asked about the language and were told about Bucharest.

One of them then switched to English and asked again. That visitor did the repair work the bot should have done, and their accommodation is in my log as another success.

The order of the fix decided whether it was honest. I raised the engine first, from seventy-three percent to ninety-eight, by writing the eleven patterns the measurement had exposed. Only then did I rewrite the sentence. Doing it the other way round would have produced a more careful claim about a bot that was just as broken.

The verb changed as well. Understand was doing work the engine has not earned, which is the same objection I make when a product calls keyword matching intelligence. It reads Romanian well enough to route you. So now it says it reads.

Cheap failures, expensive ones

The asymmetry between those two failure modes is the whole lesson, and it is not a lesson about chatbots.

A visible failure spends a little of someone's patience and buys information with it. Both parties learn something: the visitor learns where the edge is, and I get a line in the log that tells me what to write next. A confident wrong answer spends the thing I actually need, which is the visitor's belief that answers here are worth reading, and it buys me nothing, because it files itself under success.

I spent four and a half years documenting clinical trial software, where the same shape has a name and a priority. The defect that outranks the rest is the one that looks like it worked. In my own conversation design spec I wrote the rule as silent misdirection outranks visible breakage, and then I built a metric that could only see the breakage.

I would like to claim I noticed this because I am rigorous. I noticed it because I read the transcripts out of curiosity, saw a visitor type this is weird, and went looking for why. Every one of the failures in this piece was found by reading, and none of them by the number. I have also shipped documentation in my career that was fluent and wrong, which is worse than documentation that is obviously incomplete, and for the same reason: fluency suppresses the question.

What holds it now

Each failure became a test, which is the only mechanism I trust to keep a fix fixed after I have forgotten why I made it.

The identity questions now have their own topic, and the rule that routes them is written as a rule about subjects: if a sentence names me, it is asking about my work, and if it points at this conversation, it is asking about the bot. Twenty-six phrasings are pinned in the suite, including the two from that session. The reaction this is weird has a designed reply now, which owns the miss before explaining it, because reading a complaint as a question is its own insult.

The Romanian promise stopped being a sentence and became a measurement. The suite runs the visitor-style corpus on every build, fails under ninety percent, and separately forbids the word understand from coming back into that line. A claim the bot makes about itself is now something the build checks, which is the only version of that claim I am willing to publish.

On the morning I published this, the guide answered across ninety-six topics and the suite ran nine thousand eight hundred and thirty-four assertions, all passing. Both numbers have grown since, and the current ones are on the case study. That number is large because it is cheap to add cases, so it is not a coverage claim and it deserves no credit as one. What it means is narrower and more useful: every assertion in it exists because something went wrong first, most of them in a live conversation with somebody who did not get what they came for.

What I cannot see

The coverage number is still in the log, and it will still read one hundred percent on a day when the bot is confidently wrong about something I have not thought of yet. That is not a flaw I have fixed. It is the shape of the measurement, and the only detector I have for that class of failure is a person reading the transcripts, which does not scale and will not always be me.

So the practice is the finding. Read the log, not the number. Treat a metric that only reports success as a description of what you chose to count. And when the thing you built makes a promise on your behalf, measure the promise before someone else does, because a reader with one sentence and no access to my test suite found this faster than my dashboard did.

The next measurement is already obvious and I have not done it: I know how often the bot answers, and I still do not know how often it answers well. Coverage was the easy question. I chose it because it was easy, and I would rather write that down than let this piece end on the part where I look diligent.