Emotion Recognition Using Continuous Wavelet Transform and Ensemble of Convolutional Neural Networks through Transfer Learning from Electroencephalogram Signal
,
Abstract
Purpose: Emotions are integral brain states that can influence our behavior, decision-making, and functions. Electroencephalogram (EEG) is an appropriate modality for emotion recognition since it has high temporal resolution and is a noninvasive and cheap technique.
Materials and Methods:.A novel approach based on Ensemble pre-trained Convolutional Neural Networks (ECNNs) is proposed to recognize four emotional classes from EEG channels of individuals watching music video clips. First, scalograms are built from one-dimensional EEG signals by applying the Continuous Wavelet Transform (CWT) method. Then, these images are used to re-train five CNNs: AlexNet, VGG-19, Inception-v1, ResNet-18, and Inception-v3. Then, the majority voting method is applied to make the final decision about emotional classes. The 10-fold cross-validation method is used to evaluate the performance of the proposed method on EEG signals of 32 subjects from the DEAP database.
Results:.The experiments showed that applying the proposed ensemble approach in combinations of scalograms of frontal and parietal regions improved results. The best accuracy, sensitivity, precision, and F-score to recognize four emotional states achieved 96.90%±0.52, 97.30±0.55, 96.97±0.62, and 96.74±0.56, respectively.
Conclusion: So, the newly proposed model from EEG signals improves recognition of the four emotional states in the DEAP database.
1- Soraia M Alarcao and Manuel J Fonseca, "Identifying emotions in images from valence and arousal ratings." Multimedia Tools and Applications, Vol. 77 (No. 13), pp. 17413-35, (2018).
2- Ross Gordon, Joseph Ciorciari, and Tom van Laer, "Using EEG to examine the role of attention, working memory, emotion, and imagination in narrative transportation." European Journal of Marketing, (2018).
3- Colleen A Brenner, Samuel P Rumak, Amy MN Burns, and Paul D Kieffaber, "The role of encoding and attention in facial emotion memory: an EEG investigation." International journal of psychophysiology, Vol. 93 (No. 3), pp. 398-410, (2014).
4- Sebastian Schindler and Florian Bublatzky, "Attention and emotion: An integrative review of emotional face processing as a function of attention." Cortex, Vol. 130pp. 362-86, (2020).
5- John Atkinson and Daniel Campos, "Improving BCI-based emotion recognition by combining EEG feature selection and kernel classifiers." Expert Systems with Applications, Vol. 47pp. 35-41, (2016).
6- Saime Akdemir Akar, Sadık Kara, Sümeyra Agambayev, and Vedat Bilgiç, "Nonlinear analysis of EEGs of patients with major depression during different emotional states." Computers in biology and medicine, Vol. 67pp. 49-60, (2015).
7- Yingjie Li, Dan Cao, Ling Wei, Yingying Tang, and Jijun Wang, "Abnormal functional connectivity of EEG gamma band in patients with depression during emotional face processing." Clinical neurophysiology, Vol. 126 (No. 11), pp. 2078-89, (2015).
8- James A Russell, "A circumplex model of affect." Journal of personality and social psychology, Vol. 39 (No. 6), p. 1161, (1980).
9- Antonio Maria Chiarelli, Pierpaolo Croce, Arcangelo Merla, and Filippo Zappasodi, "Deep learning for hybrid EEG-fNIRS brain–computer interface: application to motor imagery classification." Journal of neural engineering, Vol. 15 (No. 3), p. 036028, (2018).
10- Fahimeh Afshani, Ahmad Shalbaf, Reza Shalbaf, and Jamie Sleigh, "Frontal–temporal functional connectivity of EEG signal by standardized permutation mutual information during anesthesia." Cognitive neurodynamics, Vol. 13 (No. 6), pp. 531-40, (2019).
11- Ahmad Shalbaf, Sara Bagherzadeh, and Arash Maghsoudi, "Transfer learning with deep convolutional neural network for automated detection of schizophrenia from EEG signals." Physical and Engineering Sciences in Medicine, Vol. 43 (No. 4), pp. 1229-39, (2020).
12- Abdulhamit Subasi, Turker Tuncer, Sengul Dogan, Dahiru Tanko, and Unal Sakoglu, "EEG-based emotion recognition using tunable Q wavelet transform and rotation forest ensemble classifier." Biomedical Signal Processing and Control, Vol. 68p. 102648, (2021).
13- Tian Chen, Sihang Ju, Fuji Ren, Mingyan Fan, and Yu Gu, "EEG emotion recognition model based on the LIBSVM classifier." Measurement, Vol. 164p. 108047, (2020).
14- KS Bhanumathi, D Jayadevappa, and Satish Tunga, "Feedback Artificial Shuffled Shepherd Optimization-Based Deep Maxout Network for Human Emotion Recognition Using EEG Signals." International Journal of Telemedicine and Applications, Vol. 2022(2022).
15- Yu-Xuan Yang et al., "A recurrence quantification analysis-based channel-frequency convolutional neural network for emotion recognition from EEG." Chaos: An Interdisciplinary Journal of Nonlinear Science, Vol. 28 (No. 8), p. 085724, (2018).
16- Hao Chao and Liang Dong, "Emotion recognition using three-dimensional feature and convolutional neural network from multichannel EEG signals." IEEE sensors journal, Vol. 21 (No. 2), pp. 2024-34, (2020).
17- Wei-Long Zheng, Jia-Yi Zhu, and Bao-Liang Lu, "Identifying stable patterns over time for emotion recognition from EEG." IEEE Transactions on Affective Computing, Vol. 10 (No. 3), pp. 417-29, (2017).
18- Yunyuan Gao, Xiangkun Wang, Thomas Potter, Jianhai Zhang, and Yingchun Zhang, "Single-trial EEG emotion recognition using Granger Causality/Transfer Entropy analysis." Journal of Neuroscience Methods, Vol. 346p. 108904, (2020).
19- Yannick Roy, Hubert Banville, Isabela Albuquerque, Alexandre Gramfort, Tiago H Falk, and Jocelyn Faubert, "Deep learning-based electroencephalography analysis: a systematic review." Journal of neural engineering, Vol. 16 (No. 5), p. 051001, (2019).
20- Yanming Guo, Yu Liu, Ard Oerlemans, Songyang Lao, Song Wu, and Michael S Lew, "Deep learning for visual understanding: A review." Neurocomputing, Vol. 187pp. 27-48, (2016).
21- Geert Litjens et al., "A survey on deep learning in medical image analysis." Medical image analysis, Vol. 42pp. 60-88, (2017).
22- Heekyung Yang, Jongdae Han, and Kyungha Min, "A multi-column CNN model for emotion recognition from EEG signals." Sensors, Vol. 19 (No. 21), p. 4736, (2019).
23- Jingxia Chen, Dongmei Jiang, Yanning Zhang, and Pengwei Zhang, "Emotion recognition from spatiotemporal EEG representations with hybrid convolutional recurrent neural networks via wearable multi-channel headset." Computer Communications, Vol. 154pp. 58-65, (2020).
24- Fangyao Shen, Guojun Dai, Guang Lin, Jianhai Zhang, Wanzeng Kong, and Hong Zeng, "EEG-based emotion recognition using 4D convolutional recurrent neural network." Cognitive Neurodynamics, Vol. 14 (No. 6), pp. 815-28, (2020).
25- Yongqiang Yin, Xiangwei Zheng, Bin Hu, Yuang Zhang, and Xinchun Cui, "EEG emotion recognition using fusion model of graph convolutional neural networks and LSTM." Applied Soft Computing, Vol. 100p. 106954, (2021).
26- Haiping Huang, Zhenchao Hu, Wenming Wang, and Min Wu, "Multimodal emotion recognition based on ensemble convolutional neural network." IEEE Access, Vol. 8pp. 3265-71, (2019).
27- Shuaiqi Liu, Xu Wang, Ling Zhao, Jie Zhao, Qi Xin, and Shui-Hua Wang, "Subject-independent emotion recognition of EEG signals based on dynamic empirical convolutional neural network." IEEE/ACM Transactions on Computational Biology and Bioinformatics, Vol. 18 (No. 5), pp. 1710-21, (2020).
28- Emad-ul-Haq Qazi, Muhammad Hussain, Hatim AboAlsamh, and Ihsan Ullah, "Automatic Emotion Recognition (AER) System based on Two-Level Ensemble of Lightweight Deep CNN Models." arXiv preprint arXiv:1904.13234, (2019).
29- Aniket Singh Rajpoot and Mahesh Raveendranatha Panicker, "Subject independent emotion recognition using EEG signals employing attention driven neural networks." Biomedical Signal Processing and Control, Vol. 75p. 103547, (2022).
30- Sara Bagherzadeh, Keivan Maghooli, Ahmad Shalbaf, and Arash Maghsoudi, "Emotion recognition using effective connectivity and pre-trained convolutional neural networks in EEG signals." Cognitive Neurodynamics, pp. 1-20, (2022).
31- Sara Bagherzadeh, Keivan Maghooli, Ahmad Shalbaf, and Arash Maghsoudi, "Recognition of emotional states using frequency effective connectivity maps through transfer learning approach from electroencephalogram signals." Biomedical Signal Processing and Control, Vol. 75p. 103544, (2022).
32- Junxiu Liu et al., "EEG-based emotion classification using a deep neural network and sparse autoencoder." Frontiers in Systems Neuroscience, p. 43, (2020).
33- Juan Cheng et al., "Emotion recognition from multi-channel eeg via deep forest." IEEE Journal of Biomedical and Health Informatics, Vol. 25 (No. 2), pp. 453-64, (2020).
34- Jiaxin Ma, Hao Tang, Wei-Long Zheng, and Bao-Liang Lu, "Emotion recognition using multimodal residual LSTM network." in Proceedings of the 27th ACM international conference on multimedia, (2019), pp. 176-83.
35- Sander Koelstra et al., "Deap: A database for emotion analysis; using physiological signals." IEEE transactions on affective computing, Vol. 3 (No. 1), pp. 18-31, (2011).
36- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton, "Imagenet classification with deep convolutional neural networks." Advances in neural information processing systems, Vol. 25(2012).
37- Yashar Taghizadegan, Nader Jafarnia Dabanloo, Keivan Maghooli, and Ali Sheikhani, "Obstructive sleep apnea event prediction using recurrence plots and convolutional neural networks (RP-CNNs) from polysomnographic signals." Biomedical Signal Processing and Control, Vol. 69p. 102928, (2021).
38- Karen Simonyan and Andrew Zisserman, "Very deep convolutional networks for large-scale image recognition." arXiv preprint arXiv:1409.1556, (2014).
39- Christian Szegedy et al., "Going deeper with convolutions." in Proceedings of the IEEE conference on computer vision and pattern recognition, (2015), pp. 1-9.
40- Sergey Ioffe and Christian Szegedy, "Batch normalization: Accelerating deep network training by reducing internal covariate shift." in International conference on machine learning, (2015): PMLR, pp. 448-56.
41- Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna, "Rethinking the inception architecture for computer vision." in Proceedings of the IEEE conference on computer vision and pattern recognition, (2016), pp. 2818-26.
42- Shaoqing Ren, Jian Sun, K He, and X Zhang, "Deep residual learning for image recognition." in CVPR, (2016), Vol. 2, p. 4.
43- Ludmila I Kuncheva, Combining pattern classifiers: methods and algorithms. John Wiley & Sons, (2014).
44- Marina Sokolova and Guy Lapalme, "A systematic analysis of performance measures for classification tasks." Information processing & management, Vol. 45 (No. 4), pp. 427-37, (2009).
45- Edmund T Rolls, "Limbic systems for emotion and for memory, but no single limbic system." Cortex, Vol. 62pp. 119-57, (2015).
46- Peter J Morgane, Janina R Galler, and David J Mokler, "A review of systems and networks of the limbic forebrain/limbic midbrain." Progress in neurobiology, Vol. 75 (No. 2), pp. 143-60, (2005).
47- Wei-Long Zheng and Bao-Liang Lu, "Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks." IEEE Transactions on Autonomous Mental Development, Vol. 7 (No. 3), pp. 162-75, (2015).
2- Ross Gordon, Joseph Ciorciari, and Tom van Laer, "Using EEG to examine the role of attention, working memory, emotion, and imagination in narrative transportation." European Journal of Marketing, (2018).
3- Colleen A Brenner, Samuel P Rumak, Amy MN Burns, and Paul D Kieffaber, "The role of encoding and attention in facial emotion memory: an EEG investigation." International journal of psychophysiology, Vol. 93 (No. 3), pp. 398-410, (2014).
4- Sebastian Schindler and Florian Bublatzky, "Attention and emotion: An integrative review of emotional face processing as a function of attention." Cortex, Vol. 130pp. 362-86, (2020).
5- John Atkinson and Daniel Campos, "Improving BCI-based emotion recognition by combining EEG feature selection and kernel classifiers." Expert Systems with Applications, Vol. 47pp. 35-41, (2016).
6- Saime Akdemir Akar, Sadık Kara, Sümeyra Agambayev, and Vedat Bilgiç, "Nonlinear analysis of EEGs of patients with major depression during different emotional states." Computers in biology and medicine, Vol. 67pp. 49-60, (2015).
7- Yingjie Li, Dan Cao, Ling Wei, Yingying Tang, and Jijun Wang, "Abnormal functional connectivity of EEG gamma band in patients with depression during emotional face processing." Clinical neurophysiology, Vol. 126 (No. 11), pp. 2078-89, (2015).
8- James A Russell, "A circumplex model of affect." Journal of personality and social psychology, Vol. 39 (No. 6), p. 1161, (1980).
9- Antonio Maria Chiarelli, Pierpaolo Croce, Arcangelo Merla, and Filippo Zappasodi, "Deep learning for hybrid EEG-fNIRS brain–computer interface: application to motor imagery classification." Journal of neural engineering, Vol. 15 (No. 3), p. 036028, (2018).
10- Fahimeh Afshani, Ahmad Shalbaf, Reza Shalbaf, and Jamie Sleigh, "Frontal–temporal functional connectivity of EEG signal by standardized permutation mutual information during anesthesia." Cognitive neurodynamics, Vol. 13 (No. 6), pp. 531-40, (2019).
11- Ahmad Shalbaf, Sara Bagherzadeh, and Arash Maghsoudi, "Transfer learning with deep convolutional neural network for automated detection of schizophrenia from EEG signals." Physical and Engineering Sciences in Medicine, Vol. 43 (No. 4), pp. 1229-39, (2020).
12- Abdulhamit Subasi, Turker Tuncer, Sengul Dogan, Dahiru Tanko, and Unal Sakoglu, "EEG-based emotion recognition using tunable Q wavelet transform and rotation forest ensemble classifier." Biomedical Signal Processing and Control, Vol. 68p. 102648, (2021).
13- Tian Chen, Sihang Ju, Fuji Ren, Mingyan Fan, and Yu Gu, "EEG emotion recognition model based on the LIBSVM classifier." Measurement, Vol. 164p. 108047, (2020).
14- KS Bhanumathi, D Jayadevappa, and Satish Tunga, "Feedback Artificial Shuffled Shepherd Optimization-Based Deep Maxout Network for Human Emotion Recognition Using EEG Signals." International Journal of Telemedicine and Applications, Vol. 2022(2022).
15- Yu-Xuan Yang et al., "A recurrence quantification analysis-based channel-frequency convolutional neural network for emotion recognition from EEG." Chaos: An Interdisciplinary Journal of Nonlinear Science, Vol. 28 (No. 8), p. 085724, (2018).
16- Hao Chao and Liang Dong, "Emotion recognition using three-dimensional feature and convolutional neural network from multichannel EEG signals." IEEE sensors journal, Vol. 21 (No. 2), pp. 2024-34, (2020).
17- Wei-Long Zheng, Jia-Yi Zhu, and Bao-Liang Lu, "Identifying stable patterns over time for emotion recognition from EEG." IEEE Transactions on Affective Computing, Vol. 10 (No. 3), pp. 417-29, (2017).
18- Yunyuan Gao, Xiangkun Wang, Thomas Potter, Jianhai Zhang, and Yingchun Zhang, "Single-trial EEG emotion recognition using Granger Causality/Transfer Entropy analysis." Journal of Neuroscience Methods, Vol. 346p. 108904, (2020).
19- Yannick Roy, Hubert Banville, Isabela Albuquerque, Alexandre Gramfort, Tiago H Falk, and Jocelyn Faubert, "Deep learning-based electroencephalography analysis: a systematic review." Journal of neural engineering, Vol. 16 (No. 5), p. 051001, (2019).
20- Yanming Guo, Yu Liu, Ard Oerlemans, Songyang Lao, Song Wu, and Michael S Lew, "Deep learning for visual understanding: A review." Neurocomputing, Vol. 187pp. 27-48, (2016).
21- Geert Litjens et al., "A survey on deep learning in medical image analysis." Medical image analysis, Vol. 42pp. 60-88, (2017).
22- Heekyung Yang, Jongdae Han, and Kyungha Min, "A multi-column CNN model for emotion recognition from EEG signals." Sensors, Vol. 19 (No. 21), p. 4736, (2019).
23- Jingxia Chen, Dongmei Jiang, Yanning Zhang, and Pengwei Zhang, "Emotion recognition from spatiotemporal EEG representations with hybrid convolutional recurrent neural networks via wearable multi-channel headset." Computer Communications, Vol. 154pp. 58-65, (2020).
24- Fangyao Shen, Guojun Dai, Guang Lin, Jianhai Zhang, Wanzeng Kong, and Hong Zeng, "EEG-based emotion recognition using 4D convolutional recurrent neural network." Cognitive Neurodynamics, Vol. 14 (No. 6), pp. 815-28, (2020).
25- Yongqiang Yin, Xiangwei Zheng, Bin Hu, Yuang Zhang, and Xinchun Cui, "EEG emotion recognition using fusion model of graph convolutional neural networks and LSTM." Applied Soft Computing, Vol. 100p. 106954, (2021).
26- Haiping Huang, Zhenchao Hu, Wenming Wang, and Min Wu, "Multimodal emotion recognition based on ensemble convolutional neural network." IEEE Access, Vol. 8pp. 3265-71, (2019).
27- Shuaiqi Liu, Xu Wang, Ling Zhao, Jie Zhao, Qi Xin, and Shui-Hua Wang, "Subject-independent emotion recognition of EEG signals based on dynamic empirical convolutional neural network." IEEE/ACM Transactions on Computational Biology and Bioinformatics, Vol. 18 (No. 5), pp. 1710-21, (2020).
28- Emad-ul-Haq Qazi, Muhammad Hussain, Hatim AboAlsamh, and Ihsan Ullah, "Automatic Emotion Recognition (AER) System based on Two-Level Ensemble of Lightweight Deep CNN Models." arXiv preprint arXiv:1904.13234, (2019).
29- Aniket Singh Rajpoot and Mahesh Raveendranatha Panicker, "Subject independent emotion recognition using EEG signals employing attention driven neural networks." Biomedical Signal Processing and Control, Vol. 75p. 103547, (2022).
30- Sara Bagherzadeh, Keivan Maghooli, Ahmad Shalbaf, and Arash Maghsoudi, "Emotion recognition using effective connectivity and pre-trained convolutional neural networks in EEG signals." Cognitive Neurodynamics, pp. 1-20, (2022).
31- Sara Bagherzadeh, Keivan Maghooli, Ahmad Shalbaf, and Arash Maghsoudi, "Recognition of emotional states using frequency effective connectivity maps through transfer learning approach from electroencephalogram signals." Biomedical Signal Processing and Control, Vol. 75p. 103544, (2022).
32- Junxiu Liu et al., "EEG-based emotion classification using a deep neural network and sparse autoencoder." Frontiers in Systems Neuroscience, p. 43, (2020).
33- Juan Cheng et al., "Emotion recognition from multi-channel eeg via deep forest." IEEE Journal of Biomedical and Health Informatics, Vol. 25 (No. 2), pp. 453-64, (2020).
34- Jiaxin Ma, Hao Tang, Wei-Long Zheng, and Bao-Liang Lu, "Emotion recognition using multimodal residual LSTM network." in Proceedings of the 27th ACM international conference on multimedia, (2019), pp. 176-83.
35- Sander Koelstra et al., "Deap: A database for emotion analysis; using physiological signals." IEEE transactions on affective computing, Vol. 3 (No. 1), pp. 18-31, (2011).
36- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton, "Imagenet classification with deep convolutional neural networks." Advances in neural information processing systems, Vol. 25(2012).
37- Yashar Taghizadegan, Nader Jafarnia Dabanloo, Keivan Maghooli, and Ali Sheikhani, "Obstructive sleep apnea event prediction using recurrence plots and convolutional neural networks (RP-CNNs) from polysomnographic signals." Biomedical Signal Processing and Control, Vol. 69p. 102928, (2021).
38- Karen Simonyan and Andrew Zisserman, "Very deep convolutional networks for large-scale image recognition." arXiv preprint arXiv:1409.1556, (2014).
39- Christian Szegedy et al., "Going deeper with convolutions." in Proceedings of the IEEE conference on computer vision and pattern recognition, (2015), pp. 1-9.
40- Sergey Ioffe and Christian Szegedy, "Batch normalization: Accelerating deep network training by reducing internal covariate shift." in International conference on machine learning, (2015): PMLR, pp. 448-56.
41- Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna, "Rethinking the inception architecture for computer vision." in Proceedings of the IEEE conference on computer vision and pattern recognition, (2016), pp. 2818-26.
42- Shaoqing Ren, Jian Sun, K He, and X Zhang, "Deep residual learning for image recognition." in CVPR, (2016), Vol. 2, p. 4.
43- Ludmila I Kuncheva, Combining pattern classifiers: methods and algorithms. John Wiley & Sons, (2014).
44- Marina Sokolova and Guy Lapalme, "A systematic analysis of performance measures for classification tasks." Information processing & management, Vol. 45 (No. 4), pp. 427-37, (2009).
45- Edmund T Rolls, "Limbic systems for emotion and for memory, but no single limbic system." Cortex, Vol. 62pp. 119-57, (2015).
46- Peter J Morgane, Janina R Galler, and David J Mokler, "A review of systems and networks of the limbic forebrain/limbic midbrain." Progress in neurobiology, Vol. 75 (No. 2), pp. 143-60, (2005).
47- Wei-Long Zheng and Bao-Liang Lu, "Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks." IEEE Transactions on Autonomous Mental Development, Vol. 7 (No. 3), pp. 162-75, (2015).
| Files | ||
| Issue | Vol 10 No 1 (2023) | |
| Section | Original Article(s) | |
| DOI | https://doi.org/10.18502/fbt.v10i1.11512 | |
| Keywords | ||
| Emotion Recognition Electroencephalogram Deep Learning Transfer Learning Ensemble Approach Continuous Wavelet Transform | ||
| Rights and permissions | |
|
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License. |
How to Cite
1.
Bagherzadeh S, Maghooli K, Shalbaf A, Maghsoudi A. Emotion Recognition Using Continuous Wavelet Transform and Ensemble of Convolutional Neural Networks through Transfer Learning from Electroencephalogram Signal. Frontiers Biomed Technol. 2022;10(1):47-56.
