CCSS: Unobtrusive Epidermal Sensors and Deep Gaussian Process Networks for Silent Speech Recognition from Articulatory Muscle Activities

Silent speech interfaces are systems that do not rely on voiced speech signals for speech communications. Such interfaces have long been pursued to help people with voice disorders caused by diseases, laryngectomy, accidents, vocal abuse, or aging. Existing silent speech systems face several challenges, including bulkiness, obtrusiveness, immobility, and poor robustness against interferences. The lack of a truly efficient and unobtrusive system has precluded its continuous use and wide-scale deployment. This project aims to develop an efficient, skin-friendly, mechanically/visually unobtrusive, and personalized silent speech interface. Skin-attachable sensors are designed to track muscle activities induced by speech. Novel machine learning methods are developed for converting the sensed signals to spoken speech and for customized silent speech recognition. The developed hardware-software platform will impact broad fields, such as rehabilitation, healthcare, motion tracking, robot control, and human-machine interactions. For example, the developed system can help restore spoken communication for people with voice disorders. The system provides a more natural input method than touch and gesture to interact with smart machines, such as cellphones, computers, prosthetics, robots, and virtual/augmented reality. The system is particularly useful when private information delivery is needed, a quiet environment is desired, or voice-based speech signals are compromised by underwater or noisy environments. In addition, emerging techniques from this project will be adopted to develop education modules, demonstrations, or lab components to enrich classroom teaching, capstone design projects, and K-12 outreach activities. The development of the proposed customizable and robust platform that captures and interprets speech-relevant muscle activities for silent speech recognition is carried out through three specific research tasks. First, unobtrusive electromyogram sensing electrodes for tracking speech-relevant muscle activities will be designed and characterized. Second, machine learning algorithms based on deep Gaussian process networks will be explored to decode sensing signals for speech recognition. Finally, machine learning methods will be developed to assist personalization and optimization of the sensing array. This project will yield new knowledge about the design of biopotential sensors with good sensing capability while addressing the issues of user-friendliness. The developed machine learning methodology is general and can easily be extended to other domains beyond speech recognition. The optimization of sensor layouts for each user will allow for customized systems that accommodate the unique muscle movement patterns of each user and maximize speech recognition accuracy. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.