Deep learning of task and resting state fMRI data

Decoding brain functional states underlying cognitive processes from task fMRI data using multivariate pattern analysis techniques has achieved promising performance for characterizing brain activation patterns and providing neurofeedback signals. However, it remains challenging to decode subtly distinct brain states for individual fMRI data points due to varying temporal durations and dependency among different cognitive processes. To overcome these limitations, we have developed a deep learning framework for brain decoding by leveraging recent advances in intrinsic functional network modeling and sequence modeling using long short-term memory (LSTM) recurrent neural networks (RNNs). Particularly, subject-specific intrinsic functional networks (FNs) are computed from resting-state fMRI data and are used to characterize functional signals of task fMRI data with a compact representation for building brain decoding models, and LSTM RNNs are adopted to learn brain decoding mappings between functional profiles and brain states [18].  For pattern recognition modeling of resting state fMRI data, functional connectivity measures have been successfully used to predict individual phenotypes. However, most existing studies focus on coarse-grained functional connectivity measures between brain regions or intrinsic connectivity networks, which may sacrifice fine-grained functional connectivity information of the fMRI data. Since whole brain voxel-wise functional connectivity measures could provide fine-grained functional connectivity information of the brain and may improve the prediction performance, we have developed a deep learning method to use convolutional neural networks (CNNs) to learn informative features from the fine-grained whole brain functional connectivity measures [19].

Schematic illustration of the proposed brain decoding framework. (a) The overall architecture of the proposed model; (b) LSTM RNNs with two LSTM layers (N=2) used in this study; (c) Data split used for model training, parameter selection, and testing; (d) Functional profile clipping for data augmentation and model training.