A/B Test Your AI Receptionist Before You Keep the Change

An AI receptionist A/B test compares saved versions against the same defined outcome. It needs trustworthy participant records before it needs a winner: Microsoft reported that about 6% of tests in one historical analysis had a sample-ratio mismatch, a warning that the observed audience split differed from the intended one. That is a measurement lesson, not an Entagl failure rate. Microsoft Research

Your new opening question may sound better in a preview. Whether it helps real customers finish a booking is a different question.

When experiments are enabled in your paid Receptionist workspace, you can compare a change against a defined business outcome. Access depends on your workspace and team permissions. The examples below are hypothetical, not customer results.

What should you test first?

Choose a decision you would actually make differently after seeing evidence. A clinic might compare asking for a preferred day first with asking which service the customer needs. A store might compare a concise product recommendation with a question that narrows the selection.

Write the hypothesis in plain words: “Asking which service the customer wants before offering times will increase confirmed bookings.” Then choose one primary outcome before the test starts. If your real goal is confirmation, do not quietly switch to message count because it looks more favorable.

Keep the rest of the experience steady. Changing the offer, model, instructions and follow-up policy together may answer whether the whole package works, but it cannot tell you which change caused a difference. Record outside changes, including a new promotion or a staff rota change, so you can interpret the result later.

Our guide to training an agent on business knowledge covers what the agent should know. An experiment asks what happens when you change how it uses that knowledge.

How does Receptionist separate the versions?

The web flow starts in the agent canvas. Open A/B tests, select the exact connected account or supported website target, then create a draft. A saves the original setup. B is an experimental copy. You can inspect the changes without assigning the draft as the live account configuration.

A test targeting one connected account is not a workspace-wide change. Check the displayed account name, handle or number, especially when several accounts share a source pipeline. Website presentation settings can be shared by several embeds, so review the affected target before applying a result.

Choose a fixed traffic split. That ratio stays fixed throughout the test; traffic does not automatically move toward the version currently ahead. Actual participant counts will not necessarily match the configured percentages exactly. Message counts are separate again, because one participant can send several messages.

The design saves the tested configuration and the assignment separately from what actually ran. If delivery fails or a fallback is used, that should remain visible in the evidence. Removing those participants would make a troubled version look better than the experience customers received.

Which test type answers your question?

Question	Test type	What stays separate
Does different agent behavior help?	Two saved pipeline versions	Follow-up policy stays the same
Does AI re-engagement help?	Follow-up on versus off	Staff messages and unrelated campaigns are not switched off
How should website chat appear?	Launcher, welcome bubble or auto-open	The chat content and offer stay fixed

These are different experiments. A test of conversation instructions does not also prove that an automatic follow-up helps. A test of an opening panel does not prove a better booking conversation.

For follow-up tests, count all assigned eligible conversations. Counting only people who received a successful send would exclude failures and change the question. Existing opt-outs, human takeover, channel limits, funding checks and AI pause rules still matter.

What makes a result usable?

A higher number on the screen is not enough. Check whether participants had time to reach the selected outcome, whether tracking stayed healthy and whether the difference is large enough to matter to your business.

Receptionist uses recorded events instead of asking an AI model to decide whether a conversation was successful. It preserves pending and unknown outcomes. A customer who has not yet completed the measurement window is not automatically a failure, and missing tracking is not a zero.

Stopping enrollment also does not instantly finish measurement. People already assigned may still complete a booking during their allowed window. The results screen should show that collection period before presenting a completed result.

Microsoft’s guidance distinguishes frequent monitoring for harmful effects from a final decision at the planned end of a test. The same distinction matters for a receptionist: stop a broken experience when necessary, but do not declare a winner because a small early lead looks attractive. Microsoft Research

How do you keep or undo a change?

Review the saved differences and the supporting conversations. In an experiment, Use as main is a deliberate action with a target check. Test history keeps the saved setup, dates, results and decision. Restoring saved A and returning to inherited routing are different choices: the first restores a specific saved version; the second follows the current parent routing rule.

An inconclusive test can stay inconclusive. It may tell you to gather more evidence in a new run, choose a larger practical change, or keep the original. It does not create an obligation to adopt B.

This builds on the controlled rollout approach in launching an AI agent without losing customers. A saved history makes the decision easier to inspect later; it is not a promise that every change can bypass current permission or dependency checks.

What happens when the test ends or evidence is missing?

At the planned end, normal routing is the default. You may instead explicitly preselect saved A, B or C where available. That scheduled choice still needs experiment availability, current target and dependency checks; it is not an automatic winner selection. A blocked choice needs attention and must not be reported as applied. Existing participants can continue through their outcome windows.

Unknown outcomes have a strict consequence: even one unresolved unknown can prevent a statistical recommendation for that run. Descriptive results and an explicit user choice remain available. A manually stopped run without a completed planned analysis horizon is not proof of a winner. On a downgrade to Free, experiment access and new enrollment close, and experiment handling is interrupted safely. Ordinary Inbox records remain.

FAQ

Is a preview conversation an A/B test?

No. A preview checks sample behavior. An A/B test assigns eligible real traffic and measures a predefined outcome. Preview messages must not count as experiment participants or business successes.

Does a 50/50 split guarantee equal message counts?

No. The split applies to eligible participants, not individual messages. Random allocation can produce unequal participant counts, and participants can send different numbers of messages. Unexpected allocation patterns need an integrity check.

Can Receptionist automatically choose the winner?

Traffic stays at the fixed ratio you choose. You decide which saved version to use, either at the scheduled end or after review. Automatic traffic adaptation and AI Manager/MCP-guided creation are not included.

Who can use the feature?

Experiments must be enabled in your paid Receptionist workspace and allowed by your team permissions. Native mobile is for results and permitted controls; authoring belongs on the web. AI usage continues to consume credits.

Bring one proposed change and one business outcome to a Receptionist discussion. Book a demo to review whether this testing approach fits your traffic and workflow.

Sources checked 10 October 2026. Microsoft’s historical figures are not a forecast for Entagl or your business. No conversion increase is promised.