How to Keep an AI Agent Accurate After It Goes Live

AI agent maintenance is not cleanup work. It is the job. An agent that passed every test at launch can get measurably worse in production without anyone touching a prompt, a guardrail, or a line of code. A peer-reviewed study in Nature's Scientific Reports paired four machine-learning models with 32 real-world datasets from healthcare operations, finance, transportation and weather, then tracked how each pairing held up as time passed since its last training. Quality decayed in 91% of the 128 pairings, a pattern the authors named AI aging. The fix is not a better model. It is a review loop someone owns.

We have written about getting an agent safely to go-live in four stages. This is the part after that, which gets far less attention and causes more quiet damage: what you do in week six, and week twenty.

Why does an AI agent get worse after launch?

Four things decay, and they decay independently. Most teams only watch one.

What decays	What it looks like	How you catch it
The model's behaviour	Same question, same prompt, a worse answer that reads as confidently as the right one	Re-run a fixed test set on a schedule and compare scores
Your knowledge	The agent correctly quotes a policy, price, or opening hour you changed last month	Date-stamp knowledge entries; review on a cadence
Your business	New service, new location, seasonal hours, a promotion that ended	Tie knowledge updates to the operational change that caused them
The scope	The agent gets asked to do longer, more involved jobs than it was scoped for	Watch handover rate and the task types arriving

The first row surprises people. Anwar Ali's HousingWire column names this behavioural drift and separates it from the data drift most teams have heard of. It is the harder of the two to catch, because nothing about a worse answer looks worse.

What does drift actually look like in a real deployment?

Anwar Ali, SVP and Head of Product Management at BSI Financial Services, published an account of exactly this in September 2026. His team ran a voice and chat agent answering borrower questions against live account data, behind compliance guardrails they had spent a long time getting right. It worked. Containment, the share of customers whose issue resolved without reaching a person, held above the industry average for months.

Then it fell by roughly eight points. Nobody noticed for weeks.

The investigation cleared the guardrails, the conversation flow, and shifts in customer behaviour. The cause was the model itself, an established one from a leading lab, quietly producing worse answers to the same categories of question. His conclusion is the part worth stealing: it was a measurement failure, not a technology failure. Containment was reviewed weekly, and a weekly lagging indicator cannot catch a slow slide until a lot of customers have already been through it.

Which numbers should you watch, and how often?

Pick a small set, put a cadence on each, and make the cadence shorter than the damage window. A metric reviewed monthly gives you a bad month before it gives you a signal. These are operating metrics rather than return metrics; for the financial side see how to measure AI ROI when most projects show none.

Metric	Cadence	What a bad move means
Handover rate to a human	Daily	The agent is refusing or failing more often, or the questions changed
First-response time	Daily	A delivery or queue problem, not a quality problem
Resolution without handover	Daily	The classic drift indicator. Watch the trend, not the day
Thumbs-down rate on AI replies	Weekly	Your team is seeing something the aggregate numbers hide
Booking or conversion rate from conversations	Weekly	The agent is answering but no longer closing
Regression test set score	Monthly, and on every model change	The model or config moved under you

First-response time deserves a place on that list even though it rarely drifts. Our own study of 32,581 conversations found that replies inside 60 seconds converted at 35.1%, against 7.1% for replies that took one to twenty-four hours. Speed is the metric an agent is bought for, so it is the one worth proving is still true.

How often should you re-test an AI agent?

Keep a fixed regression test set of real conversations with known-correct outcomes, and re-run it on three triggers:

1. On a schedule, monthly at minimum. This is what turns drift into a number instead of a hunch.
2. Before and after any model change, including one you did not initiate. If your platform moves to a newer model, that is a change to your system.
3. After any meaningful knowledge or instruction edit. A fix for one complaint often breaks an answer that was fine.

Build it from conversations you already have. Twenty to fifty cases covering your common questions, your awkward edge cases, and the handful the agent must never answer alone. Big enough to catch a real regression, small enough that you will actually run it.

Re-benchmarking which model you run on is a separate, slower cycle. Quarterly is a reasonable default, and it is much easier if your setup was never welded to one lab, which we argued in why you should not build your business on a single AI model.

What does a weekly maintenance loop look like?

Thirty to sixty minutes, same slot every week:

1. Read ten real conversations end to end. Not summaries. Pick a few flagged ones and a few at random, because the random ones show you what normal looks like.
2. Triage every thumbs-down your team logged. Sort each into one of three buckets: the knowledge was wrong or missing, the instructions were wrong, or the agent should have handed over.
3. Fix the bucket, not the conversation. A one-off correction in a single chat teaches the system nothing. Update the knowledge entry, the instruction, or the handover rule.
4. Re-run the regression set if you changed anything load-bearing.
5. Write down what you changed and why. A dated change log is what lets you answer "when did this start?" six weeks later.

Why do longer tasks fail even when the model is fine?

This is a design constraint rather than a maintenance task, but it shows up looking like decay.

An August 2026 arXiv preprint, How Fast Do Agents Rot?, measured agent reliability across nine models and 10,664 analysed trajectories. Task success follows a geometric law governed by per-step reliability, which rises with model scale but saturates well below 1 even for the strongest systems. On the genuinely agentic tool-use task, every model tested, including widely deployed proprietary ones, fell from near-perfect success to near zero within sixteen dependent steps. Degradation tracked step count, not context length.

It is a preprint, and its tasks are not customer-service conversations. The practical read still holds: reliability compounds downward with every dependent step, so an agent scoped to answer, qualify and book stands on much firmer ground than one running a fifteen-step process unsupervised. If your agent's job has quietly grown since launch, that is a scope problem wearing a quality problem's clothes.

Who actually owns this?

Usually nobody, and that is the real finding in Ali's piece. Product teams ship, engineering keeps infrastructure up, operations runs the business, and nobody is responsible for noticing that answers got worse.

Name one person, and do not hand it to a committee. They watch the daily numbers, run the weekly loop, and have the standing to say the agent is off and pull its scope back. That last part is what makes the role real. Ali is blunt about the alternative: "a reviewer who can only rubber-stamp isn't providing oversight, just its appearance."

Where Entagl fits

Anyone on your team can flag a bad AI reply straight from the unified inbox, with a note on what was wrong. Flagged replies land in a review queue with a status, so the weekly triage has a worklist instead of a memory test. Knowledge lives in one place, so fixing an FAQ, a service, or your opening hours fixes it for WhatsApp, Instagram, Messenger, Telegram, web chat, email and the API at once. Because the four agents share one brain, a correction you make for chat is also true on the next call.

Two structural pieces matter more than any single feature. Model routing is not tied to a single lab: the model behind each stage is a setting, and backup models can be configured behind it, so a vendor's behavioural change is a routing decision rather than a rebuild. And human handover is a first-class path rather than a failure state, so the number that tells you something is drifting is one you are already collecting.

Grounding the agent properly at the start makes all of this cheaper, which we cover in how to train an AI agent on your own business knowledge.

What maintenance does not fix

It does not make an agent accurate that was never grounded properly. Monitoring a badly scoped agent gives you a precise measurement of the wrong thing.

It does not eliminate hallucination. The controls that reduce it are a separate stack, covered in how to stop your AI agent from hallucinating.

And a number moving is not a diagnosis. Handover rate can rise because the agent got worse, or because you ran an ad campaign that brought in a different kind of question. Read the conversations before you change anything. That is why step one of the weekly loop is reading, not measuring.

FAQ

How often should you check an AI agent's performance?

Watch handover rate, first-response time and resolution rate daily, because these are the leading indicators of a slide. Review flagged replies and conversion weekly. Re-run a regression test set monthly and on every model change. Weekly-only review is how an eight-point drop in one financial services deployment went unnoticed for weeks.

Can an AI agent get worse if nothing changed?

Yes, for two separate reasons, and they are worth keeping apart. Behavioural drift is the one most teams miss: a hosted model can start producing worse answers to the same questions with no change to your prompts or configuration, which is what Anwar Ali documented at BSI Financial Services when containment fell roughly eight points. The other is AI aging: a fixed model gets less accurate simply because more time has passed since it was trained, which Nature's Scientific Reports observed in 91% of the 128 model-and-dataset pairings it tested. Different mechanisms, same symptom. A fixed regression test set catches both.

How do you know if it is the model or your knowledge base?

Re-run your fixed regression test set. If cases that used to pass now fail while the underlying knowledge is unchanged, the model moved. If the agent is answering confidently from information that is simply out of date, your knowledge moved. The test set is what separates the two, which is why it has to be fixed and reused rather than rewritten each time.

Who should own AI agent maintenance in a small business?

One named person, usually whoever owns the customer experience rather than whoever is most technical. The work is reading conversations, triaging flagged replies, and updating business knowledge. None of it requires engineering, and all of it requires someone with the standing to pull the agent's scope back when the numbers say so.

Does a better model remove the need for monitoring?

No. Per-step reliability improves with model scale but does not reach 1, so longer task chains still degrade, and a stronger model can still change behaviour underneath you. Better models raise the floor. They do not remove the need to watch the floor.

Start with the number you are not collecting

You almost certainly run a weekly numbers review somewhere else in this business, keep a change log for some other system, and know how to spot-check work for quality. Ali reaches the same conclusion at the end of his column: the capability is usually already there, and almost nobody has aimed it at the AI that talks to their customers.

Pick one leading indicator you are not watching daily, give it an owner, and put thirty minutes on the calendar for the weekly read. That alone would have caught the drift described here.

To see how the flagged-reply queue, shared knowledge and handover controls work on a real workspace, book a 30-minute demo and we will walk your own conversations through it.

Sources: Nature Scientific Reports, "Temporal quality degradation in AI models" (2022); HousingWire, Anwar Ali, "The silent failure mode: what happens when your AI model quietly gets worse" (September 2026); arXiv:2609.01660, "How Fast Do Agents Rot?" (August 2026); Entagl Response Velocity Study (2026). Figures verified as of September 2026.