DASHBOT GUIDE BOOK

Chatbot
Intelligence
Maturity Curve
Table of Contents

Table of
Contents

INTRODUCTION
Chatbot Intelligence Maturity Curve
STAGE 1
Adoption of Analytics-led Design
Analytics-led Design
Adoption of Analytics-led Design: The 360 Framework

STAGE 2
Smart Goals and Intelligent KPIs
SMART business Goals
The Pyramid of Goal-Setting
Mapping your Goals to KPIs
Inherent and Predictive Conversational Metrics

STAGE 3
Raising your Chatbots IQ:
The Conversational AI Flywheel
BEYOND THE CURVE
Sustaining Analytical Maturity

2
Introduction

“You have
optimized for
your limitations.”
The analytics tools you use hit a brick wall when it comes to even displaying conversational
or open ended text, a crucial component in effectively managing chatbot interactions. This
limitation results in businesses shaping user feedback to fit predetermined outcomes rather
than truly listening and responding to what their customers are saying. As a consequence, much
of the rich insights that reside within this “dark data” are overlooked, preventing businesses
from fully optimizing their chatbot strategies to enhance user satisfaction and operational
efficiency.

Dashbot addresses these challenges head-on by advancing the capabilities of traditional

analytics with its novel suite of tools that is designed for conversational analytics. By
introducing the Chatbot Intelligence Maturity Curve, Dashbot provides a structured framework
that helps businesses progress from basic data handling to deep, insightful analytics that drive
genuine engagement and strategic alignment with business goals.

3
Maturity Curve Stages

Where are you on

the Chatbot Intelligence
Maturity Curve?
THE CHATBOT INTELLIGENCE MATURITY CURVE
The Chatbot Intelligence Maturity Curve is a strategic blueprint for organizations aiming to harness the full power
of chatbot technologies. Where you are on the curve is determined by the maturity of your chatbot program. We
at Dashbot have synthesized 3 phases that will help you move along this curve regardless of where you start.
This evolutionary process starts with high-impact yet low-maturity activities that set the foundational metrics for
analytics-driven design.

1 ANALYTICS-LED 2 SMART GOALS &

DESIGN INTELLIGENT KPIS
TANGIBLE IMPACT

MATURITY OVER TIME

4
Maturity Curve Stages

STAGE 01

Analytics-led Design
Organizations must implement analytics in the initial stages of designing chatbots. Taking a data-driven approach
to designing your bot begins with analyzing and processing your historic conversation data as opposed to starting
in a bot designing tool blindly. The current approach to building bots involves a conversation designer making
assumptions on what the best flow is based on their personal understanding of the data.

When thinking about how you will design a chatbot, focus on capturing broad metrics that can instantly influence
and improve customer interactions. At this stage, represented by our 360 Bot Framework, the impact is significant
despite the maturity of the analytics being relatively low. This phase is crucial for setting up a baseline of chatbot
performance and understanding initial customer interactions.

STAGE 02

Smart Goals and Intelligent KPIs

As the organization progresses, the focus shifts towards more strategic activities that have a lower immediate
impact but are crucial for long-term success. Here, the SMART framework comes into play, guiding teams to
structure well thought out business goals, which is critical for the second part of this phase. Mapping chatbot
functionalities directly to business KPIs, ensuring that every chatbot interaction aligns with larger business
objectives.

The relevance of your optimization efforts in the subsequent phase is determined by the quality of the KPI and
Metric mapping. Understanding which metrics and signals are most relevant and having well constructed goals
go hand in hand. This stage is marked by a deliberate approach to integrate and harmonize chatbot operations
with corporate strategies, preparing for more sophisticated analytics applications.

STAGE 03

Raising your Chatbots IQ

Reaching the peak of the maturity curve, organizations are equipped to delve into high-fidelity, in-depth
conversational analytics. This advanced stage allows businesses to visualize and sculpt comprehensive user
journeys, optimizing each interaction for maximum effectiveness and engagement. At this point, the analytics
provide deep insights that enable continuous refinement and personalization of the chatbot experience, leading
to transformative business outcomes. The culmination of this maturity is not just in achieving operational
excellence but in turning every chatbot conversation into a strategic asset.

The Chatbot Intelligence Maturity Curve illustrates a dynamic pathway from foundational implementations to
strategic integrations enabled by advanced conversational analytics. By navigating this curve, organizations
unlock the potential to transform their chatbots from simple automated tools to crucial drivers of customer
satisfaction and sources of business intelligence.

5
STAGE 01

Adoption of
Analytics-led
Design: The 360
Framework
Analytics-led Design

Analytics-led Design
Starting with an analytics-led design ensures that chatbot development is driven by data, focusing on measurable
outcomes right from the beginning. This approach helps identify crucial interaction points and influence design
choices, achieving impactful results in early chatbot deployments.

When you design a Chatbot you have to ask yourself, “what conversations are my customers already having?” or
“what is the ideal conversation my user should have?” Understand that when designing something to work with
your customers at scale, tying the decisions you make to relevant data points is important.

FORTUNE 100 INSURANCE FIRM

Analytics-led Design in Action

Challenge
A fortune 100 Insurance company came to us to redesign their support and self service architecture. Their
customer journey was complex and disconnected - it spans email, chat, phone, SMS, chatbot, and
self-service website.

With no real visibility into what happens during the conversations, challenges on how to deploy a new solution
quickly lead to time pressures to effectively execute to achieve business objectives.

“There’s a lot of variation in these conversations… but Dashbot is able to derive that this was all involving
scheduling appointments, and it seems spot on.”

- Data Science Business Partner

Solution & Results

By processing their existing data and identifying patterns and themes,
we established a “50k foot view” of the Voice of the Customer

Prevented them from attempting to build a $2M custom BI tool

Reduced data science team effort by by 66%

Predicted Sentiment and CSAT supplemented customer feedback with insights into
what happened during the journey avoiding the reliance on “collective wisdom”

7
Analytics-led Design

The 360 Self Service Framework

The 360 Framework begins with Analytics-led Design as the leading concept and the first step of your journey to
build a self service system. Ingesting your relevant conversational data sources and customer interaction data
with the goal of creating a source of truth for your customer interactions is the beginning.
Processing and doing an initial scan of the data to identify the range of automation opportunities that are available
leads into deployment and monitoring phases of the 360 framework. Knowing where to focus the automation on,
you can begin to build your self service system and see how users are interacting and behaving with it.

At the early stages, it is important to optimize for user retention and engagement before thinking too deeply
about where you want your users to take their conversations. Learning to keep an open mind about user behavior
while having strong beliefs as to what conversations are the most valuable to them will help you mature and
evolve your ability to analyze customer conversations.

PHASE 01

Dashbot ROI &

Automation Assessment
Access automation opportunities with your own
conversational data

PHASE 02

Accelerate Successful
Deployment
Leverage your conversational data for better design

PHASE 03

Monitor Performance &

Continuous Improvement
Purpose built analytics for conversational data

8
Phase 1: Dashbot’s Automation and R.O.I. Assessment

PHASE 01

Dashbot’s Automation and R.O.I. Assessment

Using a tool like Dashbot, organizations can seamlessly process and analyze all of your existing unstructured
conversational data to assess automation opportunities and calculate potential R.O.I based on your historical
data.

Automation is no longer just an advantage; it’s a necessity for scaling, improving customer experience, and
optimizing operational efficiencies. However, the path to automation is fraught with challenges, including
identifying where automation can be most effective, ensuring a seamless deployment, and continuously
improving based on performance data. Starting “from scratch” with “manual review” will yield a slow arduous
process that will be based on bias assumptions.

9
Phase 2: Accelerated Successful Deployment

PHASE 02

Accelerated Successful Deployment

After we have a factual assessment of how your customers are actually speaking to your company, we can pass
that information along for you to drop your deployment time. The Implementation process of your bot is guided
by the assessment – and the popular conversations that are already happening will be the conversations you
design your bot to have. Understanding what the customer wants to self-serve and then focusing on actually
constructing the relevant intents and training data drastically accelerates the bot design process.

10
Phase 3: AI-Driven Insights and Continuous Optimization

PHASE 03

AI-Driven Insights and

Continuous Optimization
Phase 3 of the 360 framework is relevant after you deploy your first self service system, however what you need
to optimize for will change as you move along the maturity curve. At the onset of your deployment getting the
self service accurate to what users are looking for is the most important. However, as conversational patterns
and deviations become apparent and as your business goals change, the metrics you look at to analyze will
change. From day 1 of deployment, Dashbot leverages A.I. to continuously optimize your bot and measure
business impact, while integrating more conversational sources to track the entire customer journey and assess
performance across channels. We’ll cover more of how to continuously optimize your bots in later sections.

11
STAGE 02

Mapping
Smart Goals &
Intelligent KPIs
Once you’ve designed your chatbot using a data-driven approach that we mentioned in the prior section, you
need to quantify the impact & R.O.I. of this chatbot by tying it to business goals. The foundation of building
goals is cemented by how committed you are to moving towards the goal. Goals are inherently landmarks that
you move as you hit them, the goal itself is not as critical as the direction you end up moving in. However, when
it comes to building and monitoring what could be thousands of self service conversations a day, the details
and minutiae around a goal make all the difference.
Mapping Smart Goals & Intelligent KPIs

SMART
Business Goals
To navigate the nuances of digital customer engagement effectively, setting refined, actionable goals is crucial.
This necessity underscores the introduction of the SMART framework in chatbot optimization—where clarity in
objectives harmonizes with operational and strategic demands. Through Dashbot, businesses can tap into ad-
vanced analytics, transforming these defined goals into measurable outcomes that directly enhance customer
interaction and operational efficiency.

Each SMART goal is a stepping stone towards realizing

Specific a broader vision: transforming chatbots from basic
You can clearly define the path towards achieving query responders to powerful tools of customer
your goal and picture what success looks like engagement and data insight. By aligning these
goals with the next step of mapping them to KPIs,
businesses can ensure that their chatbot strategies
Measurable are not only grounded in current capabilities but are
You can track your progress in a way that motivates also geared towards future growth and refinement.
you to continue working towards your objective This approach doesn’t just address immediate
inefficiencies; it fosters a continuous cycle of
Achievable improvement and adaptation, ensuring that chatbots
evolve in step with both technological advancements
You have the necessary tools and skills to reach your and shifting customer expectations.
goal, or at least have a plan for how to attain them
Here are some examples of SMART goals for basic
chatbot optimization:
Relevant
You are focusing on an objective that makes sense
1. Reduce Escalation Rates: Decrease the number
within the broader scope and vision of your business of conversations escalated to human agents by
15% over the next six months through enhanced
chatbot response accuracy and decision-making
Time-bound capabilities.
You have a target date for deliverables to keep yourself 2. Expand Coverage of Topics Handled: Enrich
on track and create a greater sense of urgency the chatbot’s capability by adding responses for
50 new topics, identified through data analytics,
within the next four months.
3. Enhance Engagement Metrics: Increase the
average conversation duration by 20 seconds and
interaction rates by 30% in the next five months
by introducing interactive elements and richer
content formats.
4. Optimize Response Time: Reduce the chatbot’s
average response time by 40% within three
months by optimizing processing algorithms and
increasing server capacities.

13
The Pyramid of Goal-Setting

The Pyramid
of Goal-Setting
When structuring your goals, taking a top down approach will provide the most clarity. At the top of the Iceberg
we have the Business goals that you generally want to achieve, typically this involves knowledge sharing, map-
ping training goals and aligning your organizations relevant team.

Determining the themes and general user behavior necessary to progress towards your goals. Themes, in this
context, are high-level categories that you care about, like adoption, effort and satisfaction.

Metrics are the numbers you track to measure how well you’re meeting your goals. Make sure your metrics are
tied to your business objectives. Start with the goal for the business then figure out how you’re going to mea-
sure it and then focus on optimizing for them.

01 BUSINESS GOALS

Improve Digital Customer Experience by imple-

menting a chatbot as a new support channel

02 THEME & HOW TO MEASURE

Adoption: Are users engaging and returning to the bot?

Goal attainment: Are users able to self-serve?
Adoption: Are users happy and prefer the new experience?

03 METRICS IN & OUT OF BOT

Adoption: Users, engagement, retention

Goal attainment: Escalation/containment rate,
flow completion, NLP confidence
Adoption: CSAT, NPS, sentiment
Mapping Smart Goals & Intelligent KPIs

Mapping your Goals to KPIs

KPI (Key Performance Indicator) mapping plays a crucial role in aligning chatbot functionalities with overar-
ching business goals. Dashbot, with its advanced analytics capabilities, introduces a nuanced approach to KPI
mapping, from basic engagement metrics to sophisticated AI-driven insights that transform unstructured data
into actionable business intelligence.

Inherent and Predictive

Conversation Metrics
BASIC ENGAGEMENT METRICS:
INITIALLY, KPIS FOCUS ON ESSENTIAL CHATBOT FUNCTIONALITIES

Users Activity
• Users • Sessions
• Tops Users • Engagement
• Users by Time of the day • Retention
• Demographic, origin, data, behavioral • Escapation
data, or metadata • Transcriptions
• Emotion (?) • Conversation paths

Messages Quality
• Top messages in • Default fallback intent
• Top messages out • Phrase Cluster
• Message Intent/Funnel • CSAT, NPS
• Abandonment rate for each intent • Live person takeover + triggers

These KPIs assess the chatbot’s efficiency and initial user engagement, offering insights into its operational
effectiveness.

INTUIT

Optimizing for Specific Metrics

Intuit QuickBooks Assistant received over 1 million sessions per month across 50+ bots and implement-
ed Dashbot to better understand their bot metrics. This led to a significant decrease in fallback intent by
over 35% and decrease in human escalations by 57%.

15
Mapping Smart Goals & Intelligent KPIs

Advanced Metrics & Unstructured AI Insights In the advanced stage, Dashbot leverages AI to provide deeper
insights such as:

1. A.I. Predicted Rating: is a measure of user satisfaction— a better NPS score. AI predicts how a user would
rate their interaction a scale of 1-10 based on the full transcript of each conversation.
2. Customer Effort: Applies AI to predict how much effort was needed by a user to resolve their issue on a
scale of 1-5.
3. Helpfulness Score: uses AI to predict how helpful the bot was in helping the user with their stated goals
on a scale of 1-5.
4. Dashbot’s Themes & Categories: uses AI to identify the reasons why a user engaged in a conversation
with the bot based on the raw transcript data.
5. Product Identification: AI identifies products discussed, linking interactions to specific business out-
comes without libraries of key words.

16
Mapping Smart Goals & Intelligent KPIs

How are Dashbot’s Metrics Different

& Why are your KPIs Incomplete?
The truth behind the metrics you currently use today (Net Promoter Score, Customer Satisfaction, etc.) is that
they are all based on asking a customer to tell you how they feel on a numerical scale and then those numbers
are averaged and extrapolated across the business and optimized for.

Analytics for conversations has reached the point where we can see how users actually feel based on their language,
semantics and in context circumstances. The numerical representation must come after the understanding is
gained instead of pawning the process off to the user and hoping they told the truth.

1. Predicted Sentiment and CSAT vs Traditional Closed-Ended Feedback:

Traditional methods often rely on sentiment scoring based on average or aggregate values of positive
words and negative words contained within a message. CSAT is often derived by single-response surveys
at the end of a survey or customer interaction. Both of these methodologies are inflexible and lack nuance.
Human communication cannot be summed up by simply assigning values to words in a vacuum and adding
them up nor can it be boiled down to a user telling you how many stars out of 5 they felt.

In contrast, AI and LLMs delve into open-ended analysis, capturing nuanced emotional responses across
entire conversations. This approach provides a more dynamic and comprehensive understanding of
sentiment, moving beyond the limitations of binary or scaled responses to capture the full spectrum
of customer emotions. The satisfaction of the user is extracted from the feeling they capture in their
conversation and in order to achieve that the words must be taken in context of the sentence and not in the
context of all discourse across all time.

2. Themes and Categories vs. Traditional Topic Analysis:

Traditional topic analysis tends to categorize customer feedback into predefined topics based on libraries
of words. This method restricts the breadth of insights gathered and your ability to form novel insights.

Dashbot’s theme and category analysis, however, utilizes more fluid, data-driven approaches to identify
emerging themes in real-time. Opening up users to novel high and low level themes that their users
are actually talking about. This method adapts to the evolving language and expressions of customers,
uncovering deeper insights into customer needs and preferences

3. Customer Effort vs. Traditional CES (Customer Effort Score):

Traditional CES metrics often quantify effort through direct questions and taking a mass average of the
answers, which leads to skewed perceptions if the customer is influenced by recent experiences or not
reporting their true feelings.

AI enhancements evaluate customer effort more holistically by analyzing interaction patterns and the
implicit effort deduced from conversational nuances, providing a more accurate measure of the customer’s
experience.

17
Mapping Smart Goals & Intelligent KPIs

4. Customer Rating vs. Traditional NPS (Net Promoter Score):

While NPS provides a valuable snapshot of customer loyalty through its promoter, passive, and detractor
categories. Traditional NPS is calculated by asking the user if they consider themselves a Promoter or not
on a scale of 1-10 as opposed to inferring the score from the user’s conversation itself.

AI-driven customer ratings incorporate a broader range of data points. These include real-time feedback,
behavioral indicators, and contextual understanding, which is then scored based on context offering a
more layered and precise evaluation of customer satisfaction and loyalty.

The integration of AI and LLMs in analyzing these metrics represents a significant shift towards more dynamic,
accurate, and insightful customer analytics. This advanced approach not only enhances the precision of data
interpretation but also aligns more closely with the actual experiences and sentiments of customers, driving
more informed and effective business decisions.
After defining your goals and metrics, you can use a dashboard to aggregate and consolidate all this data in one
place. This will help you identify trends and make data-driven decisions about how to improve your chatbot.

18
STAGE 03

Raising your
Chatbots IQ: The
Conversational AI
Flywheel
The Conversational AI Flywheel

The Conversational AI Flywheel

Once you have a chatbot deployed and are mapping your business goals to quantifiably KPIs, how do you
continuously optimize your chatbot through an insights-driven flywheel? As chatbots evolve within the
conversational analytics workflow, the challenge of scaling effectively becomes apparent. The volume and
complexity of data make trend and pattern analysis increasingly challenging. Initially, dashboards provide a
basic level of insight, but they often fall short of explaining the “why” behind emerging trends. This is where the
maturity of chatbot analytics, guided by Dashbot’s tools, becomes crucial.

20
The Conversational AI Flywheel

Mapping Journeys and Flows

As your self service program evolves, the need to get granular with where are groups of users diverting off of the
conversation or abandoning the session. Diving into User Journeys and conversation flows is required to pinpoint
where and why users deviate from the intended interaction paths. By narrowing down data conversations to
only relevant ones that deviate or to a specific part of the ideal conversation path, teams can identify specific
users and conversations that veer off course. The Dashbot platform, with its novel User Journey visualizer, allows
teams to analyze large datasets to uncover the underlying causes of these deviations at scale and across different
customer groups or geographies as conversations may differ amongst different user groups.

21
The Conversational AI Flywheel

Normalize your Data Sources:

The Dashboard of Truth
Omni-channel dashboards are pivotal for synthesizing insights across various communication platforms and
operating as a source of truth for your data set. As businesses reach their users through multiple channels—from
social media to chat and email—the ability to aggregate and analyze data cohesively becomes indispensable.
Integrating all your chatbots and customer touchpoints provides a unified view that reveals next level user
behavior patterns. This holistic approach not only helps in identifying consistent trends but also in detecting
nuanced differences in how operational decisions impact customer behavior across different channels. By
converting raw data from these diverse sources into actionable insights, Dashbot empowers organizations to
refine their communication strategies, ensuring a cohesive and personalized user experience that resonates
across all platforms.
The 3 Why’s: Going from Information to Insights

The 3 Why’s:
Going from Information to Insights
Whether you use a 3rd-party analytics provider or an in-house solution, it likely only provides you with numbers
on a dashboard (i.e. roll-up of metrics) to answer basic questions that start with “How many…?” This type of
data is simply information, not an insight. Insights are answers to your increasingly complex questions that start
with “Why” or “How” that allow you to take action to drive your initiatives forward. As you progress through
different stages of maturity, your questions naturally become more complex and so you need additional tooling
– additional data sources, additional visualization tools, etc

INFORMATION VS. INSIGHTS

action

Dashbot’s Predicted Metrics transform chatbots from simple automated responders into strategic business
tools, enhancing user interaction quality and driving significant business growth through detailed, AI-driven
insights. When you build a trend based on basic insights, your understanding is stalled at the question, “why?”
The best you can do is mull something over in your head before attempting to read as many customer transcripts
as you can to understand the truth. We have the technology to read customer interactions, identify trends and
understand why all at scale, so that you can focus on synthesizing the information as opposed to processing it.

23
The 3 Why’s: Going from Information to Insights

A critical part of the chatbot optimization process is gathering these insights and taking action on them to
continuously optimize your bot model. Using a tool such as our Phrase Clusters below, you can quickly go from
Insights to Action by understanding what utterances need to be added or optimization in your chatbot model to
improve the customer experience quickly.

24
The 3 Why’s: Going from Information to Insights

BEYOND THE CURVE

Sustaining Analytical Maturity

The Evolution of Conversational Analytics
The journey through the Chatbot Analytics Maturity Curve is not just about enhancing technological capabilities
but also about embedding deeper strategic insights into every conversation. By meticulously mapping KPIs to
business goals and advancing through the stages of analytics maturity, organizations can transform their chatbots
from simple automated responders to powerful tools of business intelligence and customer engagement.

As chatbots continue to evolve, they become pivotal in sculpting not only how businesses communicate with their
customers but also how they adapt to and anticipate customer needs, ensuring that every chatbot interaction
adds value to both the user experience and the business.

DROPBOX

From Raw Transcripts to Actionable Insights

Dropbox came to us with an already mature support and self-service program and difficult challenge; performance
was beginning to stagnate and the team was struggling with an increasingly high volume of service requests. They
relied on inefficient and manual processes for analyzing the bot’s customer journey, resulting in time-consuming
drawing and manual reviewing of data.

“Dashbot is key for us to both report outcomes to the business and prioritize where to dedicate resources
to enhancing our customer journey.”

— Maureen O’Sullivan, Program Manager

The collaboration between Dropbox and Dashbot

yielded impressive results:
Achieved a 43% decrease in Increased self-service adoption for Expanded self-service automation to more evolve into
unresolved requests in the bot, even “Account upgrade” topics by 3x, handling increasingly complex service topics such as,
with a 15x increase in chat volume. leading to decreased service costs. “implementation” within one year.

Dashbot’s advanced machine learning automatically processes raw chat transcripts and enriches it in Data
Slicer to signal and identify top customer support requests and reveal commonly asked questions. By visualizing
customer journey flows and tying the insights into the Omni-Channel Dashboard, Dropbox redesigned its bot and
self-service experience to deliver proactive digital service.

25
Conclusion

CONCLUSION

Overcoming your
limitations drives
your evolution
Navigating the Chatbot Intelligence Maturity Curve transforms foundational chatbot
interactions into strategic business assets. Starting with Analytics-led Design, companies
establish a baseline in the 360 Self Service Framework for your self service chatbot, which
evolves through precise KPI mapping in the SMART framework.

Choose the right metrics to optimize for and the right dimensions to view for signals &
patterns. As your program matures, continuously review these metrics and dimensions. This
progression culminates in Stage 3, where you can leverage advanced conversational analytics
to optimize your ideal user journeys, blend in additional data, and maintain omniscience with
your conversational dashboard.

As AI-driven chatbots navigate the complexities of human dialogue, they not only serve but
also question the underlying patterns of our communications. They are mirrors reflecting what
people need and want. Technologically overcoming our limitations is one half of the journey.
Looking at conversations with fresh eyes to see the previously elusive possibilities is the other
half. We must rework the systems that have been built around our limitations to evolve our
abilities and our heuristics.

Visit www.dashbot.io to discover how to

elevate customer experiences to the next level