Exploiting Deep Learning for Sentence-Level Lipreading

Lipreading, also called visual speech recognition, is an excellent technique to understand what a speaker says without audios. Based on deep learning, many studies have gained outstanding achievements in English lipreading. English is dominated by polysyllables, with the proportion of homonyms as low as 1%. Compared to English, Mandarin (the official language of Chinese) is dominated by monosyllables, with the ratio of homonyms as high as 72%. The high ratio of homonyms makes the lipreading in Mandarin much more challenging. However, little attention has been paid to Mandarin lipreading within the deep learning framework, especially at the sentence level. In this paper, we first introduce a dataset, named Mandarin-Lipreading, which is recorded in the controlled lab environment and is the largest dataset so far for sentence-level lipreading in Mandarin. We further investigate the modeling units and propose an end-to-end Mandarin lipreading system. The experimental results show that the proposed system achieves 14.77% CER on Mandarin-Lipreading and 8.7% CER for unseen speakers evaluation on the GRID corpus.