GestureVoice: Enabling Multimodal Text Editing for Blind Users Using Gestures and Voice

Figure 1:GestureVoice enables screen-free text editing for blind users.Text editing on smartphones presents substantial difficulties for blind users, particularly in mobile situations where using the smartphone touch screen is challenging. While voice input allows for hands-free text creation, editing the text typically requires physical interaction with the touchscreen, negating the benefits of the hands-free input mechanism. This paper introduces GestureVoice, a novel multimodal approach that enables screen-free text editing for blind users. By leveraging smartwatch-based hand gestures for navigation and voice commands for correction, GestureVoice allows users to edit text without any contact with their smartphones. GestureVoice replaces cumbersome screen-based interaction for choosing the navigation granularity with an intuitive mid-air hand gesture. It also introduces an adaptive crown cursor (rotating the physical dial of the watch) to smoothly navigate to the edit location. A preliminary study highlighted the significant time spent by blind users correcting text errors using traditional methods. In contrast, our evaluation with 8 blind users demonstrates that GestureVoice achieves a 53.80% reduction in text editing time, offering a more efficient, intuitive, and screen-free solution for blind users.