Module 1: Automatic Speech Recognition(ASR).
1.1 ASR with BERT based Spell Corrector.
1.2 Self Supervised Knowledge Distillation framework for ASR.
Module 2: Visual Speech Recognition(VSR).
Module 3: Audio Visual Speech Recognition(AVSR).
Copyright Approved on Automated Captioning for E-Learning Contents for Hearing Impaired Using Cross modal Transformer based Audio and Visual Fusion Framework.
3.2 Intermediary Level AVSR Fusion
Module 4: Knowledge Distillation Framework for Audio Visual Speech Recognition.
Copyright Applied on Cross Modal based Knowledge Distillation Framework for Audio Visual Speech Recognition

• Tokenization and word embeddings are created for the ASR output sequence.
• The tokens are then sequentially examined with a spell error detector, to identify the misspelled tokens.
• Masking is applied on the misspelled tokens.
• BERT contextually finds the corrected words using 12 multi head attention layers.
• Inference: Improves the performance of ASR model with reduced WER.

• This method allows for efficient self supervised learning; first, pre-train the model on a large quantity of unlabeled speech, then fine-tune on a smaller labeled dataset.
• We have conducted experiments on three ratios as (60:40, 70:30, and 80:20)

• Convert the video into frames/images.
• Perform face detection and lip localization to find the region of interest using 68 shape predictor method.
• Implement the encoder part which is composed of multi head attention, feed forward neural network and ReLu as hidden layer activation function.
• The output context vector and the previous layer output are fed into the decoder part which is composed with softmax activation as output function to predict the current phoneme output.
• The decoded graphemes are applied CTC loss function to remove the duplicates.

• The method uses a teacher student technique to train a Visual Speech Recognition model from an Audio Speech Recognition model, with the goal of improving lip reading performance by pre training on huge unlabeled datasets.
• Knowledge distillation is the process of model compression that involves teaching a smaller network, one step at a time, exactly what to do with a larger, already trained network.