Towards Designing a Context-Aware Multimodal Voice Assistant for Pronoun Disambiguation: A Demonstration of GazePointAR

Voice assistants (VAs) like Siri and Alexa have transformed how humans interact with technology; however, their inability to consider a user's spatiotemporal context, such as surrounding objects, drammatically limits natural dialogue. In this demo paper, we introduce GazePointAR, a wearable augmented reality (AR) system that resolves ambiguity in speech queries using eye gaze, pointing gesture, and conversation history. With GazePointAR, a user can ask "what's over there?" or "how do I solve this math problem?" simply by looking and/or pointing. We describe GazePointAR's design and highlight supported use cases.