Beyond-Voice: Leveraging Articulatory Motion for Next-gen AI-Assistants

Current Human-AI speech interfaces are limited to the acoustic signal, overlooking the rich information in articulatory motion. This paper explores how complementing the acoustic channel with signals from speech articulators can enhance Human-AI interaction. Captured via a range of sensors (e.g., IMU, EMG, or cameras), these non-acoustic signals offer a new layer of insight. These signals enable applications infeasible with audio alone, such as anticipating a user’s cognitive struggle before they speak, maintaining a silent channel for private human-AI interaction, providing real-time speech coaching, and even monitoring well-being by capturing subtle cues like jaw clenching. The integration of these “beyond-voice” interfaces with large language models presents an opportunity to move from voice-only interactions to a more complete model of human-AI communication. We outline the potential research directions and the associated challenges the research community must address to realize this vision, followed by key takeaways from our exploratory study.