What Does a Conversational Product Team Actually Do?

“The importance of updating a bot is similar to the importance of updating a website.”

Hundreds of questions were submitted by attendees of Botmock’s AMA about Techniques for Conversational Design and Development. Five industry experts with backgrounds in conversation design answered as many as we could in a live roundtable discussion. To register for future (free!) AMA sessions and watch the full recording of this session, you can do so here.

The following conversation design experts joined us to answer your questions:

Maaike Groenewege, Conversation designer & coach @ Convocat

Amanda Stevens, Director of Conversation Design @ Master of Code Global

Sarah Storm, Head of Ideation & Conversation Design @ BOOM Integrated

Anna Mackenzie, Software Developer @ Myplanet

Brenda Martins, Conversational Designer @ LogMeIn

In Part I, we covered the following topics:

• Is a conversational flow ever finished? Should it always be iterated upon? How do you know when it’s done?
• How do you balance the business goals of your company? Like customer acquisition or conversion with designing empathetic conversations?
• How do you source and manage data of natural conversations?
• I often find that writing short descriptions or guidance (necessary for conversational interactions) takes me a long time because I’m never sure if it’s correct. Does the panel also struggle with this?

Q1: Is a conversational flow ever finished? Should it always be iterated upon? How do you know when it’s done?

Brenda

In reality, you are never done. However, you do get to the point where tuning your application will no longer affect the metrics in a significant way. With that said, even when the application is working very efficiently, it is advisable to periodically review the system and make sure that it still is meeting the goals, that the scope hasn’t changed, and that the answers to the questions haven’t changed. So yes, even if the bot is excellent, a periodical review is good practice.

Amanda

The importance of updating a bot is similar to the importance of updating a website. Like any other digital asset, it should be periodically reviewed and adjusted for relevancy and fine-tuning. Regarding how to know if the conversational design is finished and ready for launch, I believe we can ask ourselves a couple of questions: Does it finish the task that it was designed for? Does it do everything you want it to do? Has the design team studied all the edge cases that could occur? Is there error handling at every step of the way? If a user falls out of the conversational flow, do they have the ability to get back in?

Aside from all those considerations, it is very important to conduct user testing, and if we have ticked all the boxes there too I would say that the conversational flow is ready for launch. After that we keep a watchful eye and that will determine all the necessary fine-tuning.

The importance of updating a bot is similar to the importance of updating a website.

Sarah

From a creative perspective It can be hard to let go, but keeping in mind that “perfect is the enemy of done” helps me stop the endless adjusting and tweaking and encourages me to be willing to put it out there, get it tested and see what happens from there.

Anna

The primary purpose of designing something is the user’s fulfillment. So, if the user is reaching what we have defined as fulfillment, we have our error handling dialed, and the response catches are sorted for a particular iteration of an intent, then the conversational flow of that particular matter could be considered done. With that said, a conversational design is never done, since the designers will become aware of more changes they want to make once the conversational flow is being put to the test by live user interactions.

Maaike

It is important to maintain both the conversational flow and the responses to make sure they are accurate and up to date. If there’s one thing that COVID-19 has shown us, it’s that user trust is really dependent on how up to date the information is. For example, right now in regards to the pandemic, there are changes in facts and procedures almost every day, so it is imperative to have a team that is monitoring the latest developments and constantly updating the information that’s presented in your chatbot or voice assistant.

Q2: How do you balance the business goals of your company? Like customer acquisition or conversion with designing empathetic conversations?

Brenda

To keep a holistic perspective on the business goals we must know our best practices and commit to a course of action and the desired result. For example, we might enter into a conversation with the marketing team regarding some wordy marketing jargon that they want to add to the conversation; if as a designer you advise against their wishes, it is important to find a middle ground that is grounded on research and convincing data. Some of this is just having soft skills and building a relationship with the stakeholders. To achieve compelling design there must be trust across departments. Ideally, the team should trust that your intentions are in the best interest of the product, their goals and that your design will ultimately push towards their roles. As designers, sometimes we can let it go, but sometimes it’s a “No, that’s against best practices. Let’s do it this way”. It is good to know which battles to pick and to have alternatives in place and be able to show that if done in an alternative way, we’re going to get the desired results.

Amanda

Having a clear philosophy and defined goals helps navigate these questions. At Master of Code, our business goal is to create amazing conversations, so we must trust that our designers want to design conversational flows that promote retention and containment. At our company, we believe empathy is an intrinsic element of compelling conversational design. All conversational experiences should offer a level of empathy. If we want to present our customers with a human-like look and feel, empathy is very important. People are more likely to speak to bots that can demonstrate empathy.

Sarah

At our company, we are working to make sure that we create personas and a conversational flow that feels natural. We design to make sure that our language is appropriate, empathetic, and that the bot doesn’t ask for things that we can hold on to. We most recently built for kids and had to stay mindful of COPPA regulations

At work, I’m part of a team of four people that all handle a different piece of the pie. We are in close contact and we’re able to prep our partners for what to expect. When four people are working on one goal, everyone has to be willing to compromise a little bit. With that said, creating empathetic conversations is something that the whole team must agree upon and strive to. If the conversation doesn’t feel natural, and in our case, enriches the story so it provides a pleasurable playing experience, then it doesn’t serve our goals and our client’s goals.

Q3: How do you source and manage data of natural conversations?

Amanda

This is different for every business. At Master of Code, we store conversations and use that data to create documents that Business Analysts, Bot Tuners, and Conversational Designers can review. Our team uses Google Drive to collaborate. We regularly extract real user conversations from our NLP engines and put them in what we call user input reports. These reports allow us to see all the unprompted inputs that a user provided to the bot. The information obtained from these reports gives us great insight: is the bot responding as it should? Are there opportunities for new intents? What is the confidence score and should the bot be adjusted and tuned in any way?

Overall it is good practice to evaluate conversations and have them reviewed by multiple members of the team. Conversational Designers, Business Analysts, and Bot Tuners can all provide different insights that will improve the quality of a conversational experience. So keep it collaborative, pull data regularly, present it in a format that is easy for others to see, and make those actionable decisions.

Anna

There are two parts to this question. There’s the analytics aspect, which is what the user is saying, and then there is what the system is saying which are the phrases and intents that were written on the script for the bot to say.

Any conversational experience will be supported by a database and a set of tools that you and your developer are using. One common thing that people are using as a database and prototyping tool is Google Sheets. With that said, I do not recommend using a spreadsheet for a production level voice action or skill because your data is subject to corruption. For databases, I recommend using a CMS (content management system) that will ensure that your conversation flow cannot be edited when it is live.

I do not recommend using a spreadsheet for a production level voice action or skill because your data is subject to corruption.

The second part of this question relates to managing data that you receive from what people say to the bot. There are a lot of analytical tools that help with this and Dashbot is a great example of this type of tool. A tool like this one will provide you with a comprehensive profile on each user as well as the recording of their conversation with the bot.

Q4: I often find that writing short descriptions or guidance (necessary for conversational interactions) takes me a long time because I’m never sure if it’s correct. Does the panel also struggle with this?

Maaike

Yes, I do recognize this from my practice, and there are three things that I try to do when I’m stuck in a rut:

First, I always try to write with a fellow writer or someone that I can soundboard with. I’ll read out my dialogue/content and hear how it sounds; this technique usually gets me going again. The second thing I do is make sure that I know who the owner of the information is from a business perspective; that way, when I make things more conversational, I don’t lose the meaning of what I’m trying to communicate. And finally, I apply several techniques from technical communication and instructional design that give me a frame of mind and a feeling of how to write this kind of short, service-oriented, and problem-solving copy.

I apply several techniques from technical communication and instructional design…

Sarah

Working with a team is very helpful and prototyping with tools that let us do constant audible revisions helps too. Listening to the conversational flow helps determine the spots where the copy has gone awry. Personally, when I’m thinking about my guidance, I take some principles from writing for audio, some podcasting lessons, and I design with a mind-frame that everyone is different in the way they acquire auditory information. In my opinion, prompts should consist of a maximum of three sentences, four words each, simple phrasing with very few clauses. I trust that if I’m following those principles and working it out with someone, I should reach a successful outcome.

…prototyping with tools that let us do constant audible revisions helps…Prompts should consist of a maximum of three sentences, four words each, simple phrasing with very few clauses.

Sometimes I like to create a draft with everything that I want to say and then edit out all of the nonessential bits; after that, I add a touch of flavor for the persona and then test it out.

Brenda

This is a great issue to discuss. For example: how do you take a complex answer and distill it down into something simple enough to put in an interface where the cognitive load is an issue? I also like to roleplay the design with stakeholders and anybody who will listen.

I have to insert tech support topics into my bots pretty frequently, and the strategy I use is based on trying to hit the high points while also providing some resources for the rest. For example, if a user types in the bot: “why can’t I connect?”. Since that intent might have 12 different answers I’m not going to troubleshoot a dozen things. So in this case, we try to start by providing information for the most common fix.

“Did that work?” Bot says

“No.” says the user

“Okay, I have a couple other things to try.” Says the bot.

At this point suggest the top two or three other fixes. If those suggestions don’t work then provide a link to some detailed troubleshooting. So, hit the high points, hit that 80/20 rule and see if you can find the small bit that might help the most and then hand off the rest to the tech support channel.

Hit the high points, hit that 80/20 rule and see if you can find the small bit that might help the most and then hand off the rest to the tech support channel.

To stay updated and register for future (free!) AMA sessions, you can do so here.

The Botmock team is excited that more of our community members are involved than ever before. 🤗

This article is an expansion of the first topic covered in Botmock’s recent AMA about becoming a conversation designer. You can also read recaps of the other two sections, The foundations of conversation design and Companies are investing in conversation designers.