Loading...
Please wait, while we are loading the content...
Similar Documents
Lipreading Architecture Based on Multiple Convolutional Neural Networks for Sentence-Level Visual Speech Recognition
| Content Provider | MDPI |
|---|---|
| Author | Jeon, Sanghun Elsharkawy, Ahmed Kim, Mun Sang |
| Copyright Year | 2021 |
| Description | In visual speech recognition (VSR), speech is transcribed using only visual information to interpret tongue and teeth movements. Recently, deep learning has shown outstanding performance in VSR, with accuracy exceeding that of lipreaders on benchmark datasets. However, several problems still exist when using VSR systems. A major challenge is the distinction of words with similar pronunciation, called homophones; these lead to word ambiguity. Another technical limitation of traditional VSR systems is that visual information does not provide sufficient data for learning words such as “a”, “an”, “eight”, and “bin” because their lengths are shorter than 0.02 s. This report proposes a novel lipreading architecture that combines three different convolutional neural networks (CNNs; a 3D CNN, a densely connected 3D CNN, and a multi-layer feature fusion 3D CNN), which are followed by a two-layer bi-directional gated recurrent unit. The entire network was trained using connectionist temporal classification. The results of the standard automatic speech recognition evaluation metrics show that the proposed architecture reduced the character and word error rates of the baseline model by 5.681% and 11.282%, respectively, for the unseen-speaker dataset. Our proposed architecture exhibits improved performance even when visual ambiguity arises, thereby increasing VSR reliability for practical applications. |
| Starting Page | 72 |
| e-ISSN | 14248220 |
| DOI | 10.3390/s22010072 |
| Journal | Sensors |
| Issue Number | 1 |
| Volume Number | 22 |
| Language | English |
| Publisher | MDPI |
| Publisher Date | 2021-12-23 |
| Access Restriction | Open |
| Subject Keyword | Sensors Dentistry and Oral Surgery 3d Densely Connected Cnn 3d Multi-layer Feature Fusion Cnn Convolutional Neural Network Deep Learning Lipreading Speech Recognition Visual Speech Recognition |
| Content Type | Text |
| Resource Type | Article |
