why developers stopped trusting it
The first version drafted with the same confidence whether it was right or wrong. So developers stopped trusting the agent even when it was right.
how does an agent earn the right to act on someone's behalf, and know when to step back?
i evaluated it on its own production record
I ran three methods on the agent's complete production record: every pull request, issue, and release it made on real open-source repositories, met by real maintainers.
Expert inspection against an explicit rulebook. Rule by rule: does each claim trace to its source, does the tone fit the register, is the structure complete, is the terminology consistent.
The agent's complete production record, each failure coded into a class and pinned to the artifact where it happened. For a niche product, this is the entire population of its output.
the fine-tuning loop
Findings fed the system back: the style guide, the prompts, and the QA gates were tuned, and the loop reran on fresh output. An instrument different from user testing.
seven ways it broke
Every failure in the corpus was coded into a class. Drag through them.
Many small PRs, global context lost
Source of truth split across two trackers
Dead links after deleted releases
A stalled PR that nobody decides on
Fragile links across repos and forks
A refusal that offers no alternative
A refusal born of domain misreading
failures became rules
Three of the seven classes had no rule in the style guide. Now they do. Open each to see the rule it earned.
the ruleA master issue acts as the table of contents; work ships in linked batches.
the ruleReleases never get deleted. They get marked deprecated, so links keep resolving.
the ruleEvery stall gets a named next step and a decision deadline.
let it decide when to act, and when to step back
The agent carries : a sense of how sure it is. One behaviour, three postures. Scroll to watch it change what it does with its own confidence.
it acts, and says so
When the change is unambiguous, the agent makes the edit and tells you plainly what it did. No hedging, no permission theatre.
it proposes, and names the one blocker
When it is not certain, it proposes and waits, naming the single thing that would settle it. One blocker up front, the rest one click away.
it stops, and hands the decision back
When the call is beyond it, it stops and hands the decision to a human. Knowing the edge of your competence is part of being trusted.
I updated the setup guide to match your new dev script and pushed it to this branch.
what I addressed, and what I did not
One graduated behaviour, not a rule per error.
I resisted writing a rule for every failure in the record. One graduated behaviour handles the whole class, so the fix scales past the cases that prompted it.
One blocker up front, not every doubt at once.
Raising every doubt at once was honest and paralysing. One blocker up front, the rest one click away.
Left out on purpose: a rule per failure, and an A/B on tone.
Without the traffic to power it, a number would only look like evidence.
every question hides a second intent, or a mood
The code says what the system is. What the reader wants, the agent has to read from the question. Read each one, then reveal what it means.
"Why is this timeout set to 30 seconds?"
They want the reason because they are about to change it. Answer both.
"I could throw this repo out the window."
A feeling in a figure of speech. It must never trigger an action.
Intent fit is its own QA gate: a "how do I" answered with an explanation fails it.
the recorded baseline, then the redesign
The before is real: a v1 run recorded on a live repository, none of my documents in the loop. The after is the same first-contact turn, redesigned from what the failure-mode analysis found. Drag the slider to rewrite it.
what the trust layer changed
Adoption recovered once the system started visibly under-claiming. Technical accuracy and drafting speed improved with it, but the result I am proudest of is softer.
Developers started reading the single-blocker message instead of re-checking the source themselves. That is the moment an agent earns delegated trust, and it was the whole goal.
what I took from it, and what's next
An agent earns delegated trust the moment it is allowed to under-claim, not when it sounds smart. The voice can never promise more certainty than the system actually has.
Demo videos of the two recorded runs, side by side, land on this page next, so the before and after can be heard as well as read.
If you are hiring for work like this, I am at write@elizamarin.com.