Test Your AI Customer Service Assistant

You can test your AI assistant the way a customer would before it answers real people: Maya, the AI business manager inside Entagl, writes test questions from your business and your real chats, runs them, and grades every reply against rules you can read. One wrong answer costs more than it seems. In a Gartner survey of 3,566 customers, published in September 2026, only 27% of customers said they would try a company's chatbot again after a negative experience, and Gartner's advice to service leaders was to "prioritize reliability over reach."

This guide covers what a test is, where the questions come from, how replies are graded, what a run costs, and what Maya does when a test fails.

What is a test in Maya?

A test is a short conversation a customer might start, plus the rules a good reply has to follow. It holds one to six customer messages and any of these checks:

Rule	Example
Must mention	"Gives the price of a first consultation"
Must not say	"50% off"
Should ask for	"The customer's preferred day"
Hand over to a person	Yes, no, or either
Guidance	"Friendly, no more than three sentences"
Approved answer (optional)	A reply you've already said is right

Each test plays out on one channel, WhatsApp unless you choose another, so the reply is judged the way that customer would see it.

Tests never reach a customer. Actions your AI would take, like a booking, are simulated and nothing is saved. Only the search through your knowledge files runs for real, so the test checks what your assistant would actually find.

Where do the test questions come from?

You can write tests yourself, but you don't have to. Ask Maya and she writes them from one of two sources:

Typical questions (the default): what customers of your kind of business usually ask, based on your own services, prices and FAQs.
Your real conversations: a sample of your most recent chats from the last 7, 30, 90, 180 or 365 days.

She can also turn replies you disliked, and customer ratings of three stars or less, into tests. Where two pieces of feedback disagree, she follows the newest and marks that test for you to review. Emails, phone numbers, links and handles are removed before anything becomes a test.

You: Create 10 tests from my real conversations from the last 30 days and run them.

She writes up to 20 tests at a time, and a workspace can keep up to 100. You can add or edit your own in Maya's settings under Test cases, or just ask:

You: Add a test: if someone asks for a discount, never say "50% off" and hand them to a person.

How is each reply graded?

Some rules don't need an AI to judge them, so they don't use one:

Put the exact words of a "must not say" rule in quotes. If they appear in the reply, the test fails.
Whether the assistant handed the chat to a person is read from what it actually did, not from what it wrote.

An AI judge scores the rest. For a rule to count as met, the judge must point to what the reply actually said, and Maya checks that the quote is really there. When the judge isn't sure, the result is inconclusive, never a pass. That's deliberate: a test that quietly passes when it shouldn't is worse than one that asks you to look.

What does a test run cost?

Small runs start right away. Anything bigger gets a quote first:

A run of five tests or fewer starts straight away.
A run of more than five comes with a credit quote first, and starts only after you say yes.
Writing new tests is quoted first too.

Each run charges what it really used, and a run stops starting new tests once it has used three times the estimate. During setup, your first runs are free (up to 40 tests, once per workspace), and Maya drafts 8 to 12 sample questions there. She sums them up in the chat and sends them as a PDF if you'd like the details (how setup works).

Can tests run on their own?

Yes, in two ways.

After a change. When Maya changes an FAQ, a service, a price or a business detail for you, she reruns the tests that change touches. This is on by default, and it runs on its own for up to five tests and 10 credits a day. Changes to your instructions or tone, and bigger checks, ask you first.

On a schedule. Pick a rhythm and a time:

You: Run my tests every Monday at 9.

You can choose every day, every weekday, weekly, or monthly (on day 1 to 28). Maya quotes the cost per run and per month, and the schedule starts after you agree. If you later add tests and a run would cost more than half again what you agreed, that run is skipped with a note instead of charging you more. A run is also skipped, with a note, when you're out of credits.

Each run ends with a report in your Maya chat, with a PDF, and a notification in the app and on your phone (and on WhatsApp, if you've connected it).

What happens when a test fails?

Tap Ask Maya to fix on the failed result. Maya opens a chat about that result and looks for the cause, using the exact version of the test that was graded:

If your assistant got it wrong, she proposes the change: an instruction, an FAQ, a price. It comes to you as an approval card, like any change in change how your AI assistant replies, just by asking.
Once you approve, she offers to rerun just that test.
If the test itself was wrong (your business changed, or the rule was too strict), she offers to change the test instead.

Nothing is changed without your OK, unless you've set Maya to Just do it mode. For a wrong answer a real customer already got, the steps are in why did my AI say that?

Where can you run tests?

Run and manage tests from the Maya chat on the web or in the Entagl phone app, from Test cases in Maya's settings, or by texting Maya on WhatsApp.

What are the limits?

Messaging only. Tests check your messaging assistant (Receptionist). Tests for voice calls aren't supported yet.
Not free after setup. Runs use credits, quoted first above five tests.
The judge can be unsure. Some results come back inconclusive. Open them and decide.
No alarm on failure. A failed scheduled run shows up in its report, not as an urgent alert.
A test is a sample, not a guarantee. It checks the questions you test. Keep adding tests from real conversations as your business changes. For the wider rollout plan, see how to launch an AI agent without losing customers.

Start with ten. Text Maya on WhatsApp and say "write 10 tests from my last 30 days of chats".

FAQ

Does testing send anything to my customers?

No. Tests run in a sandbox. Nothing is sent to anyone, and actions like bookings are simulated, not saved.

Can Maya fix a failed test on her own?

Not by default. She finds the cause and proposes a fix, and it comes to you as an approval card unless you've switched her to Just do it mode. After you approve, she can rerun the test.

How many tests should I start with?

Ten to twenty is a good start: Maya writes up to 20 at a time. Runs of five or fewer start without a quote, and a handful of key questions on a daily schedule costs little. Maya quotes the schedule once when you set it up.

Can I test my voice assistant this way?

Not yet. Test cases cover your messaging assistant. Your voice assistant has its own hands-on tester during setup, and our guide to testing an AI phone agent covers how to check one.

More Maya guides

Sources: Gartner, "Gartner Survey Finds Only 27% of Customers Would Try a Chatbot Again After a Negative Experience", September 2, 2026. Product details checked November 2026.