Interpretable Speech Emotion Recognition: A Comparative Study of BiLSTM Temporal Attention and Transformer-Based Multi-Head Self-Attention

Abstract

Speech Emotion Recognition (SER) is an important area of affective computing that enables machines to understand and respond to human emotions. However, many deep learning approaches that achieve high accuracy provide limited insight into how predictions are made, which reduces their practical reliability in sensitive domains such as education and healthcare. This study presents a comparative analysis of two attention-based models for SER using the RAVDESS dataset: a Bidirectional Long Short-Term Memory (BiLSTM) network with temporal attention and a Transformer model with multi-head self-attention. Acoustic features were extracted using 40 Mel-Frequency Cepstral Coefficients (MFCCs) together with their first- and second-order derivatives, forming a 120-dimensional input feature vector. Both models were trained and evaluated on identical data splits using accuracy, precision, recall, and F1-score. The BiLSTM with temporal attention achieved an accuracy of 70.14% and F1-score of 68.76%, outperforming the Transformer model, which recorded 51.39% and 48.30%, respectively. Attention weight analysis showed that the BiLSTM model concentrated more effectively on emotionally relevant segments of speech, improving interpretability and performance. The findings suggest that incorporating temporal attention provides a better balance between recognition accuracy and model transparency, supporting the development of reliable and explainable SER systems for real-world human–machine interaction.