Multi-Modal Repairs of Conversational Breakdowns in Task-Oriented Dialogs
Overview
Published full paper:
Recognition:
Best Paper Award at UIST 2020
Project Summary
A major problem in task-oriented conversational agents is the lack of support for the repair of conversational breakdowns. Prior studies have shown that current repair strategies for these kinds of errors are often ineffective due to: (1) the lack of transparency about the state of the system’s understanding of the user’s utterance; and (2) the system’s limited capabilities to understand the user’s verbal attempts to repair natural language understanding errors.
This paper introduces SOVITE, a new multi-modal (speech + direct manipulation) interface that helps users discover, identify the causes of, and recover from conversational breakdowns using the resources of existing mobile app GUIs for grounding.
SOVITE displays the system’s understanding of user intents using GUI screenshots, allows users to refer to third-party apps and their GUI screens in conversations as inputs for intent disambiguation, and enables users to repair breakdowns using both natural language and direct manipulation on these screenshots
Key Words
Conversational interfaces; Conversational breakdown; Chatbots; Grounding in communication; Breakdown repair; Disambiguation; Instructable agents; GUI semantics
Paper major contributions
1. A new multi-modal approach that allows users to discover, identify the causes of, and repair conversational breakdowns caused by natural language understanding errors in task-oriented agents using the GUIs of existing mobile apps.
2. The implementation of SOVITE system: an implementation of the above approach, along with a user study evaluating its effectiveness and usability.
My personal contribution
I contributed as the second author initiating this project and work from the concept ideation, literature review, error taxonomy, current error-handling methods analysis, dialog flow design, interaction design, and user testing. I worked closely with my mentor Toby Li (Computer Science Ph.D. ), who worked on the technical implementation of the whole system.
Case study
If you would like to know more details about this project or discuss it with me, feel free to contact me.