AI agents for customer support · Part 11 of 14
Rolling out in stages: from suggestions to autonomy
What to count while the agent only suggests, what to count once it acts alone, and how to widen it one step at a time, with a written reason to step back.
The short version
Give the agent more one ticket reason at a time, against numbers you write down before you start. While it suggests, count how often your team sends its draft, or approves its action, exactly as proposed. Once it acts alone, count what comes back: customers who return, mistakes found in a weekly sample, and any money given outside your rules. Then widen one thing at a time. Four steps:
Step 1Write the gates down first
Part 1 sets out the three stages: suggest, act alone on the simplest tickets, then widen. Before the first one, write down for each ticket reason what moves it on, and what sends it back. A gate set after the results are in is easy to move.
1 Suggest
It proposes each refund; a person approves.
↓ Move on after 100 proposals: at least 97 approved as proposed, and none outside your rules.
2 Alone
It refunds up to €30 by itself.
↓ Move on after four weeks: fewer than 3 in 100 back within 7 days, no mistake in the weekly sample, and no refund outside your rules.
3 Wider
Up to €40: a new version of the rule.
Step back
Any refund outside your rules sends refunds back to stage 1.
Step 2While it suggests, count what people change
In the first stage every draft passes a person, so every draft is a test case you didn’t have to write. For each reason, count the drafts sent unchanged, edited, or rewritten. For actions, count those approved as proposed, approved with a change such as another amount, or declined. Then read the edits: a missing fact points to a connection (part 6), a wrong answer to an article (part 4), a wrong amount to a rule (part 8).
Intercom’s help for Human in the Loop (in beta), where Fin asks a teammate before it replies, sets the same kind of gate: “If your team approves most drafts without changes, lower the threshold one step so Fin answers those conversations itself, and check again two weeks later.”
Intercom help, “Ask a teammate for help with Fin’s replies (Human in the Loop)”Check one setting there first: with a time limit set, if nobody answers in time, Fin sends its draft as written. Count those apart, since nobody approved them. In Zendesk’s auto assist, each suggested reply and action waits for an agent’s approval, unless an admin pre-approved that action in a procedure, so pre-approve one action at a time as it passes its gate, starting with lookups such as an order’s status.
Step 3Once it acts alone, count what comes back
Once the agent sends on its own, nobody reads a reply before the customer does. So each week, for each reason, count what comes back: customers who return within seven days, mistakes in a sample a person reads, and any refund or credit outside your rules. Part 12 shows how to count all three.
| Third week alone | This week | Gate |
|---|---|---|
| Customers back within 7 days | 2.4% | under 3% |
| Mistakes in a sample of 50 | 1 | none |
| Refunds outside your rules | none | none |
One missed gate is enough: refunds stay at this stage until the wrong amount is traced and fixed.
In software, oversight changes shape as trust grows. Among users of Anthropic’s coding agent, newer users turn on full auto-approve in about 20% of sessions, and by 750 sessions in over 40%. As Anthropic puts it: “effective oversight doesn’t require approving every action but being in a position to intervene when it matters.”
Anthropic, “Measuring AI agent autonomy in practice”, February 2026Step 4Widen one thing at a time
Then widen one thing at a time: a new ticket reason, a higher limit, a new channel, longer hours, or a larger share of customers. If two change in the same week and customers start coming back, you can’t tell which change did it. And when a gate fails, step back only what failed.
Two changes in week 3
Email and refunds start together, and customers coming back rise from week 4.
Which one did it? You can’t tell.
One change at a time
Email in week 3 changes nothing. Refunds start in week 5, and customers coming back rise from week 6.
The refunds did it. Only refunds step back.
Use your tool to hold the rest steady while you widen one thing. In Intercom, put Fin live for one customer segment or region, and add the next only when the first passes its gates; on the phone, raise Fin Voice’s share of calls a step at a time, inside and outside office hours separately. In Gorgias, add a rule that keeps VIP customers with your team, and list the topics AI Agent should hand to people.
Next in this series
Part 12: measuring an AI support agent. See all 14 parts.
Sources
- Intercom help, “Ask a teammate for help with Fin’s replies (Human in the Loop)”, read on 21 October 2026.
- Intercom help, “Live testing Fin with your team or customer segments”, read on 21 October 2026.
- Intercom help, “Deploy Fin Voice”, read on 21 October 2026.
- Zendesk help, “About auto assist”, read on 21 October 2026.
- Gorgias, “Use rules with AI Agent”, read on 21 October 2026.
- Gorgias, “How Gorgias’s AI Agent works”, read on 21 October 2026.
- Anthropic, “Measuring AI agent autonomy in practice”, 18 February 2026, read on 21 October 2026.