AI agents for customer support · Part 12 of 14
Measuring an AI support agent
How to count what the agent resolved the same way every month, whatever your vendor calls a resolution: hand-overs by reason, customers who come back, and the mistakes a count can’t see.
The short version
Helpdesk vendors count a resolution in different ways: who decides, after how long, and out of which conversations. So keep one count of your own. Of the conversations the agent took, take away the ones handed to a person and the ones whose customer came back within seven days. Count the hand-overs by reason, since each reason points to a different fix. And read a sample of the resolved ones every week, because a wrong answer the customer accepted still counts as resolved.
What each vendor counts
Vendors differ on who decides that a conversation was resolved, how long they wait, and what happens if the customer comes back. Here are four.
Intercom Fin
- Resolved when
- the customer confirms, or leaves without asking for more help
- Decided after
- an inactivity timeout
- If they come back
- deducted, if they return to the same conversation
Zendesk
- Resolved when
- an AI model reads the conversation and confirms the request was resolved
- Decided after
- messaging: 2 hours by default, up to 72; email: 72 hours
- If they come back
- email: the status can’t be reversed
Gorgias
- Resolved when
- the customer doesn’t need a human agent
- Decided after
- 72 hours from the interaction
- If they come back
- not automated, if a person is needed in those 72 hours
Ada
- Resolved when
- relevant, accurate and safe, with no hand-over
- Decided after
- chat: 24 idle hours; email: 72 hours
- If they come back
- not stated; your own feedback doesn’t change the label
The base differs too. Intercom works its outcome rates out “as a percentage of all conversations in the selected period, not only the conversations Fin was involved in”. So don’t set one vendor’s rate beside another’s, or beside last year’s if the definition changed: Zendesk introduced its tiers in May 2026. If you’re still choosing, ask each vendor how it counts (part 3).
One count of your own
Keep one count of your own, worked out the same way every month from your helpdesk’s data. Start with the conversations the agent took. Take away the ones handed to a person, and the ones whose customer came back about the same thing within seven days, on any channel. What’s left, the agent resolved. Seven days is an example: choose a window, write it down, and keep it.
Ada puts the question to its own customers: “Can you differentiate between a customer who ended a conversation with your AI Agent because they were satisfied, as opposed to one who got frustrated and gave up?”
Ada, “Resolution rate optimization”A customer who gave up in the chat may write an email the next day, so match by customer and order across channels. In Zendesk, add Explore’s “Tickets reopened” metric, filtered to end users as the updater, then match new tickets to earlier ones on the requester and the order. In Intercom, pull each conversation’s Fin resolution state through the API (conversations since August 2023) and apply your own seven-day window.
Hand-overs, by reason
A hand-over rate alone says little, because some hand-overs are your rules at work. Count them by the reason logged with each one (part 9): each points somewhere different.
In Intercom, the Fin report’s filter “Fin AI Agent: configuration based escalation reason” makes the first split for you: use it “to isolate escalations that happen due to configuration rather than a customer explicitly requesting a teammate”.
What the count can’t see
A wrong answer the customer accepted still counts as resolved in your own count, and can in a vendor’s. Only reading finds it. Each week, read a sample of the conversations counted as resolved, 50 for example, and score them as in part 10. And check every refund and credit the agent gave against your rules (part 8): a count of resolutions says nothing about the money.
Next in this series
Part 13: raising CSAT with an AI support agent. See all 14 parts.
Sources
- Intercom help, “Fin AI Agent outcomes”, read on 21 October 2026.
- Intercom help, “Fin AI Agent reporting”, read on 21 October 2026.
- Zendesk help, “About automated resolution tiers”, read on 21 October 2026.
- Zendesk help, “Why are late customer replies not added to email AI agent conversation logs after 72 hours?”, read on 21 October 2026.
- Zendesk help, “Explore recipe: Reporting on the number of reopened tickets by end users”, read on 21 October 2026.
- Gorgias, “How you’re billed for using Gorgias”, read on 21 October 2026.
- Ada, “Resolution rate optimization”, read on 21 October 2026.