What Are Chatbot Success Metrics?

Senior Content Marketing Specialist

Tomislav Krevzelj

Senior Content Marketing Specialist

Chatbot success metrics help you understand whether a chatbot is supporting your objectives and successfully managing the tasks you designed it to handle.

We’ve distilled our knowledge and experience in AI chatbots, keyword chatbots, and more into three chatbot success metrics pillars to help you measure and improve your chatbot performance.

The three pillars of chatbot success metrics

Analytics gives you insights into chatbot performance, as well as valuable customer insights.

Knowing which chatbot success metrics to look for will help you identify what your chatbot is doing well and where it can perform better. These are the metrics that help you unearth those improvement opportunities:

1. Chatbot user metrics

Chatbots are there to help you help your customers – in this case, your users. This set of metrics will show you how helpful your chatbot is by analyzing the following information:

Number of total users

The number of total users engaging with your chatbot shows how many of your users are actually using your chatbot. If this number is significantly lower than the number of customers or users you know that you have or were planning to engage with, consider the following:

  • Chatbot awareness: This often gets overlooked. Customers need to know that they can chat with your business. Make sure your bot is either visible to them on your website, add a link in confirmation emails, or provide them a scannable QR code in-store. You can also run a social media campaign to expand your reach and target specific audiences to increase chatbot adoption even further.
  • Channels: When it comes to channels, customer preferences are as unique as your customers are. Make sure your chatbot is available over a channel that is practical for your customers to use. For example, our Answers platform covers WhatsApp Business, Facebook Messenger, Viber, Live Chat, SMS, and RCS for this reason.
Chatbot success metrics: compare chatbot users with total users

Number of new users

This metric can be indicative of how well your bot solves customer enquiries. The closer the number of new users is to the number of total users engaging with your chatbot, the greater the likeliness of first-contact resolution.

Protip: You can automate lead generation throughout the sales process by deploying a chatbot to convert site visitors into quality leads.

If the ratio of new users to total users is low, however, then this could be indicative of return users whose issues were not resolved. This requires attention, since it could potentially lead to lower customer satisfaction.

Chatbot success metrics: look for a high new vs total chatbot users ratio

Bounce rate

The bounce rate metric is indicative of how useful customers find your chatbot. A high bounce rate, meaning a chatbot session that is ended before programmed completion, can indicate that users do not find the chatbot useful and prematurely end the session.

High bounce rates should be investigated and further analyzed to see where in the conversation customers are dropping off – and why.

Bounce rate is a useful chatbot success metric for indicating how useful customers find your chatbot

What exactly constitutes a high bounce rate will vary from chatbot to chatbot, depending on predefined goals that your chatbot is designed to guide users towards. The lower the bounce rate is, then the higher the chance of having a higher goal completion rate.

Goal completion rate

This chatbot success metric is the most important success indicator in the user metrics, since it shows how many users successfully completed the goals you set for your chatbot to meet.

These goals will be different for different chatbots, of course – but seeing how many users really found the closest ATM to their location or how many left their contact information – shows how well your chatbot performs.

A low completion rate, which is best defined by considering your own business goals, means you should probably look at user chats and conversations metrics to see where drop off is happening.

Completion rate shows how many users completed the goals you set for your chatbot

2. Chatbot conversation metrics

Conversation metrics show you what happens in each conversation. This gives you insight into how effective your chatbot is at resolving customer enquiries.

Session duration

While there is no ideal time, this metric is indicative of how well your chatbot performs at engaging users. The longer the session duration, the better your chatbot is at creating an engaging conversational experience.

Consider user metrics when analyzing session duration. For example, if your goal is to create a rich conversational experience with chatbots, then longer session durations with low bounce rates and higher goal completions are indicative of success.

On the other hand, long session durations with higher bounce rates and low goal completion rates could mean your chatbot is taking users along for long, fruitless conversations that end in frustration for your customers.

Session duration is a chatbot success metric that shows how well your bot performs at engaging users

Chatbot to agent takeover

This metric shows how many times an agent needed to take over a conversation from a chatbot, or how many times a user asked to be transferred to a human agent.

This indicates how successful a chatbot is at handling conversations with customers – the lower the rate, the better.

Since one of the key benefits of having a chatbot is to free up agents to focus on more complex tasks, a high rate of agent takeovers is potentially cause for concern. This is where you should analyze customer chats to see where and why chats are being transferred.

Agent takeover is a chatbot success metric showing how successful your bot is at handling customer conversations

This isn’t to say that the number of agent takeovers should be zero – the ideal contact center should be set up for seamless live agent takeovers – but keeping an eye on this metric is important to constantly develop your chatbot for high customer satisfaction.

3. Chatbot CSAT metrics

Customer satisfaction metrics are the best indicator of how well your chatbot is performing its main tasks – which are related maintaining and improving customer satisfaction.

Net Promoter Score (NPS)

Asking customers to rate their level of satisfaction after their customer service experience provides you with valuable, direct user feedback.

First-contact resolution, fast resolution, as well as a pleasant and respectful experience that reflects your brand voice are all components of experiences that result in high customer satisfaction.

If you’re getting a low NPS from customers over your chatbot, analyze poorly rated chats looking for reasons why users are leaving low ratings – an unsatisfied user will almost always share exactly what bothered them.

Illustration of how you can use a chatbot to gauge customer satisfaction and gather insights into your low NPS score

Retention rate

Satisfied customers will be repeat customers – and you can check how many customers are coming back to engage with your chatbot.

Mobile users can be identified by their International Mobile Subscriber Identity (IMSI), so you can easily see how many repeat users engage with your chatbot.

Repeat users, goal completions, and bounce rates are valuable chatbot success metrics, especially when looked at together

Analyzing this metric by referencing user and conversation metrics will show whether your repeat users are having positive chatbot interactions. If the metrics indicate repeat customers are having positive interactions, then congratulations on developing a successful chatbot and keep up the good work.

Related reading

This piece outlined the fundamentals of chatbot success metrics. Do check out the following posts to further improve your chatbot performance:

Jul 8th, 2020
5 min read
Senior Content Marketing Specialist

Tomislav Krevzelj

Senior Content Marketing Specialist