Conversation Design for Chatbots: A Framework for Educational Engagement

Client
AskMeno

Year
2022- Present

Role
Sole Conversation Designer —> Conversation Design Lead —> Head of Content and Design

Tools
User Research + Testing: Usability testing, interviews, in-class observations

Design: Internal database, Miro, Google Workspace

Analytics: Built-In Analytics + Tableau

Challenge

Create a conversational product using existing collegiate-level technology to make it appropriate for early learners.

Outcome

A collection of 25 interactive chatbots featuring virtual characters that guide young learners through educational stories, prompting questions, and fostering language comprehension and social skills.

Design Process

TL;DR: Operating in an agile organization, we used the design process to design 25 chatbots, each programmed with 300+ answers and roughly 5000+ utterances from real kids to create story-based conversational paths that help kids move from lower-order to high-order thinking.

Empathize:

  • User Research: Conduct surveys, interviews with educators, in-class observations, and review expert literature to understand the developmental stage and learning needs of children aged 4-6.

  • Educational Standard Exploration: Dive deep into the educational standards and requirements for young learners.

Define:

  • Synthesis of Findings: Analyze research insights and synthesize them to identify key user pain points, goals, and opportunities.

  • Learning Objectives Identification: Define key learning objectives and educational standards to be incorporated into chatbot conversations.

  • Scope Definition: Outline project scope and design requirements to guide development.

Ideate:

  • Scenario Brainstorming: Collaboratively brainstorm scenarios conducive to teaching identified learning objectives effectively.

  • Conversation Flow Design: Design conversation flows and interactions to engage young learners and facilitate language comprehension.

  • Innovative Approaches Exploration: Explore innovative approaches to interactive storytelling and educational content delivery.

Prototype:

  • Sample Dialog Crafting: Craft sample dialogues tailored to chosen scenarios.

  • Early Prototyping: Develop early prototypes of chatbot conversations for iterative testing and refinement.

  • Virtual Character Integration: Integrate virtual character interactions to encourage student engagement and active participation.

Test:

  • Product Testing: Conduct in-class product testing sessions with children to evaluate the effectiveness of interactive storytelling content.

  • Usage Data Analysis: Analyze product usage data and user transcripts to understand user engagement and experience.

Iterative Development:

  • Feedback Incorporation: Incorporate feedback from testing sessions and usage data analysis to make improvements.

  • Regression Testing Innovation: Implement innovations in regression testing and database management to fine-tune utterances and support early learners effectively.

  • Continuous Refinement: Iterate on chatbot conversations based on ongoing user feedback and analytics insights.

SUCCESS METRICS

  1. Engagement Metrics:

    • Interaction Rate: Measure the frequency and depth of interactions between users and the character. This can include the number of questions asked, responses provided, and overall engagement time.

    • Session Duration: Gauge the average duration of user sessions with the character. Longer sessions may indicate higher engagement and interest in the content.

    • Repeat Usage: Track the percentage of users who return to engage with the character multiple times. Repeat usage indicates sustained interest and perceived value.

  2. Effectiveness Metrics:

    • Learning Outcomes: Assess the impact of chatbot interactions on learning outcomes, such as improvements in language comprehension, critical thinking skills, and retention of educational content.

    • User Satisfaction: Gather user feedback through surveys or interviews to measure satisfaction with the chatbot experience. Positive feedback indicates that the chatbot is effectively meeting user needs and expectations.

  3. Usability Metrics:

    • Error Rate: Monitor the frequency of errors or misunderstandings during chatbot interactions. A low error rate indicates that the chatbot is effectively communicating information and responding to user queries.

    • Ease of Use: Assess the perceived ease of use of the chatbot interface through usability testing or user feedback. Intuitive design and clear navigation contribute to a positive user experience.

  4. Accessibility Metrics:

    • WCAG Compliance: Ensure that the chatbot meets accessibility standards outlined in the Web Content Accessibility Guidelines (WCAG). Conduct regular audits to verify compliance and identify areas for improvement.

  5. Technology Metrics:

    • Intent Accuracy: Measure how accurately the Natural Language Understanding (NLU) model categorizes user intents and maps them to the correct workflows. High intent accuracy ensures that the chatbot effectively understands and responds to user queries.

    • Confidence Score: Assess the confidence level of the NLU model in predicting user intents, typically represented as a percentage. A high confidence score indicates that the model is confident in its predictions, while a lower score may signal uncertainty and the need for further refinement.

  6. User Feedback:

    • Qualitative Feedback: Gather qualitative feedback from users through surveys, interviews, or direct feedback channels. Insights from user feedback provide valuable context and actionable insights for improving the chatbot's design, content, and overall user experience.

Ongoing improvements

  • Improving the usability of our product by children (not teacher-led)

  • Continuously improving accessibility based on user feedback and testing

  • Improve NLP accuracy through continued user testing, analysis, and iteration

RESOURCES

Some of my favorite resources that helped aid my team and me in this project.

Books:

  1. "Conversations with Things" by Diana Diebel and Rebecca Evanhoe

  2. "Design for Kids" by Debra Levin Gelman

  3. “Story” by Robert McKee

  4. “Screenplay” by Syd Field

  5. Designing Voice User Interfaces

  6. “Content Design” by Sarah Winters

  7. "From Solo to Scaled” by Natalie Marie Dunbar

Websites and Online Resources:

  1. Cathy Pearl’s Recs - www.cathypearl.com/faq-1#faq-conversation-design

  2. Soapbox | A beginner’s guide to designing voice experiences for kids - www.soapboxlabs.com/resource/beginners-guide/

  3. Nielsen Norman Group (NN/g) - www.nngroup.com

  4. Web Accessibility Initiative - www.w3.org/WAI/standards-guidelines/wcag/

  5. UX Collective - www.uxdesign.cc

  6. Women in Voice - https://womeninvoice.org/