A novel Object categorization decoder from fMRI signals using deep neural networks
,
Abstract
Purpose: Understanding neural mechanisms is critical for discerning the nature of brain disorders and enhancing treatment methodologies. Functional Magnetic Resonance Imaging (fMRI) plays a vital role in gaining this knowledge by recording various brain regions. In this study, our primary aim was to categorize visual objects based on fMRI data during a natural scene viewing task. We intend to elucidate the challenges and limitations of previous models in order to produce a generalizable model across different subjects using advanced deep learning methods.
Materials and Methods: We've designed a new deep learning model based on transformers for processing fMRI data. The model includes two blocks, the first block receives fMRI data as input and transforms the input data to a set of features called fMRI space. Simultaneously a visual space is extracted from visual images using a pre-trained inceptionv3 network. The model tries to construct the fMRI space similar to the extracted visual space. The other block is a fully connected (FC) network for object recognition based on fMRI space. Using transformer capabilities and an overlapping method, the proposed architecture accounts for structural changes across different voxel sizes of the subjects' brains.
Results: A unique model was trained for all subjects with different brain sizes. The results demonstrated that the proposed network achieves an impressive similarity correlation between visual space and fMRI space around 0.84 for train and 0.86 for test dataset. Furthermore, the classification accuracy was about 70.3%. These outcomes underscored the effectiveness of our fMRI transformer network in extracting features from fMRI data.
Conclusion: The results indicated the potential of our model for decoding images from the brain activities of new subjects. This unveils a novel direction in image reconstruction from neural activities, an area that has remained relatively uncharted due to its inherent intricacies.
2. Lin, T.-Y., et al. Microsoft coco: Common objects in context. in Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. 2014. Springer.
3. Du, B., et al., fmri brain decoding and its applications in brain–computer interface: A survey. Brain Sciences, 2022. 12(2): p. 228.
4. Cox, D.D. and R.L. Savoy, Functional magnetic resonance imaging (fMRI)“brain reading”: detecting and classifying distributed patterns of fMRI activity in human visual cortex. Neuroimage, 2003. 19(2): p. 261-270.
5. Haxby, J.V., et al., Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science, 2001. 293(5539): p. 2425-2430.
6. Huang, W., et al., Long short‐term memory‐based neural decoding of object categories evoked by natural images. Human Brain Mapping, 2020. 41(15): p. 4442-4453.
7. Huth, A.G., et al., Decoding the semantic content of natural movies from human brain activity. Frontiers in systems neuroscience, 2016. 10: p. 81.
8. Huth, A.G., et al., A continuous semantic space describes the representation of thousands of object and action categories across the human brain. Neuron, 2012. 76(6): p. 1210-1224.
9. Mitchell, T.M., et al., Predicting human brain activity associated with the meanings of nouns. science, 2008. 320(5880): p. 1191-1195.
10. Song, S., et al., Comparative study of SVM methods combined with voxel selection for object category classification on fMRI data. PloS one, 2011. 6(2): p. e17191.
11. Wang, C., et al., ‘When’and ‘what’did you see? A novel fMRI-based visual decoding framework. Journal of Neural Engineering, 2020. 17(5): p. 056013.
12. Haynes, J.-D. and G. Rees, Predicting the orientation of invisible stimuli from activity in human primary visual cortex. Nature neuroscience, 2005. 8(5): p. 686-691.
13. Kamitani, Y. and F. Tong, Decoding the visual and subjective contents of the human brain. Nature neuroscience, 2005. 8(5): p. 679-685.
14. Reddy, L., N. Tsuchiya, and T. Serre, Reading the mind's eye: decoding category information during mental imagery. Neuroimage, 2010. 50(2): p. 818-825.
15. Postle, B.R., The cognitive neuroscience of visual short-term memory. Current opinion in behavioral sciences, 2015. 1: p. 40-46.
16. Gaziv, G., et al., Self-supervised natural image reconstruction and large-scale semantic classification from brain activity. NeuroImage, 2022. 254: p. 119121.
17. Lin, S., T. Sprague, and A.K. Singh, Mind reader: Reconstructing complex images from brain activities. Advances in Neural Information Processing Systems, 2022. 35: p. 29624-29636.
18. Shen, G., et al., End-to-end deep image reconstruction from human brain activity. Frontiers in computational neuroscience, 2019. 13: p. 21.
19. Gu, Z., et al., Decoding natural image stimuli from fMRI data with a surface-based convolutional network. arXiv preprint arXiv:2212.02409, 2022.
| Files | ||
| Issue | Articles in Press | |
| Section | Original Article(s) | |
| Keywords | ||
| fMRI; deep learning; object recognition; ROI connectivity; brain decoding | ||
| Rights and permissions | |
|
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License. |
