How to learn from AI pilot projects quickly – part two

In this second part we look at how to build, run and assess AI pilot projects. Read part one here. Commercial AI Analyst Abbie Heslop looks at the challenges of running a proof of concept AI project, and how to extract the maximum business benefits and learning from it.

16/07/2019, updated 21/12/2020

Building and running your bot

It’s often thought that the performance of an AI chatbot or voice bot is only ever as good as the data on which it was trained. The more data you have, so the thinking goes, the better. This misconception probably comes from stories in the popular press about how many cats a Google AI has to look at before it learns to recognise them for itself.

Certainly, volume of data helps – but only if it is relevant data. To build a chatbot ideally you want access to the transcriptions of lots and lots of chat sessions between customers and live chat agents. For a voice bot you want transcriptions of call recordings.

In chatbot development we talk about intents (what customers will ask the chatbot to do) and utterances (the words customers will use to express their intents). If you have access to recordings and transcriptions of live agent chat sessions or calls then not only can you use these to train your bot, you can also do a frequency analysis to determine the business impact of automating any given intent.

Training and data

But what do you do if you don’t have access to recordings, or if all your data is in a format that you can’t easily use? The answer is to take a reasonable guess. A small group of agents will give you a very good idea of the common expressions customers use to invoke particular intents. Use this information to start training a Minimum Viable Assistant (MVA); a bot that can perform well enough that users can see its usefulness and potential. Then continue to train it with live interactions post-launch. This process ensures we can capture training data from the bots real interactions as soon as possible.

Even if we use perfect transcriptions and chat logs, these may still differ dramatically from real conversations with the bot. For example, one client’s bot had been trained on call centre transcripts prior to launch. We then had to quickly train it to cope with emojis, as the demographic using the bot in its launch location in a mobile app, was different to the person calling the call centre.

You can even, with a chatbot, initially have live agents ‘pose’ as the chatbot until you have gathered enough examples of real customer expressions to train your bot.

Test and test again

For an AI pilot project we suggest testing a bot on a small, controlled scale that nonetheless allows you to get enough data quickly. For example, you might make a chatbot available to only 10% of a website’s visitors, or a subset of customers calling the contact centre.

During a 2 to 3-month pilot project, we tend to continuously train the bot on the conversations it has as it goes and release a new iteration of the bot every two weeks so that the improvement in performance can be tracked.

In this Proof of Concept you can clearly see the number of supported, and successful conversations approaching the total number of conversations as the time passes.

As you go you can also analyse the types of queries that customers are asking the bot. You can always add in additional intents as you go and train the bot on the relevant expressions.

What success looks like

During an AI pilot project it is important to set the expectations of senior managers and other operational teams beyond the pilot team, and ensure continuous reporting of progress towards a well-defined goal.

Budget holders can kill a proof of concept project before it gets anywhere by starving it of the time, money and attention it needs to succeed. Or we have seen organisations ‘throw a problem over the wall’ to a supplier and wait for them to solve it.

To keep everyone informed and the project on course we recommend fortnightly reports and an end of AI pilot project report. These should focus on answering a few key questions that help you prove your initial hypothesis. For example:

  • What % of users engage with the bot? How many hang up, click away, or request to speak a live agent?
  • What % have a successful conclusion for each use case / topic?
  • What are customers asking that the bot can’t do? What intents and entities should we train the bot to answer?
  • Can you identify the wider business benefits such as reduction in call volumes?

Remember the primary goals of proof of concept projects are to learn something and prove something. You can now go back to your starting hypothesis and modify it based on what you have proved. The beauty of this method is that you now have a new hypothesis which can be the basis of your future pilot projects on your incremental steps to your ultimate objective.

As a final point, it’s important that everyone has realistic expectations about the results. If you’re building, say, a chatbot to automate five types of customer queries and it ends up doing those very well, then your project has succeeded within that limited domain. On occasion we have heard executives who weren’t heavily involved say that’s all very well, but it can’t answer all these other questions our customers might ask it. Well, of course it can’t at the moment – that’s the next project!

Want the ‘Ultimate Guide to Successful AI Projects for CX Leaders?’

In the ultimate guide we examine 5 common mistakes identified from pilot projects of major banks, airlines and contact centres, how to avoid them and how CX and contact centre leaders can make the most of the technologies that are going to define the next generation of customer interaction.