|
Cerebral Cortex 大脑皮层, Volume 32, Issue 14, 15 July 2022, Pages 2972–2984, ，第32卷，第14期，2022年7月15日，第2972-2984页,https:///10.1093/cercor/bhab394 Published: 返回文章页面 16 November 2021 2021年11月16日 Abstract摘要Limited sample size hinders the application of deep learning in brain image analysis, and transfer learning is a possible solution. However, most pretrained models are 2D based and cannot be applied directly to 3D brain images. In this study, we propose a novel framework to apply 2D pretrained models to 3D brain images by projecting surface-based cortical morphometry into planar images using computational geometry mapping. Firstly, 3D cortical meshes are reconstructed from magnetic resonance imaging (MRI) using FreeSurfer and projected into 2D planar meshes with topological preservation based on area-preserving geometry mapping. Then, 2D deep models pretrained on ImageNet are adopted and fine-tuned for cortical image classification on morphometric shape metrics. We apply the framework to sex classification on the Human Connectome Project dataset and autism spectrum disorder (ASD) classification on the Autism Brain Imaging Data Exchange dataset. Moreover, a 2-stage transfer learning strategy is suggested to boost the ASD classification performance by using the sex classification as an intermediate task. Our framework brings significant improvement in sex classification and ASD classification with transfer learning. In summary, the proposed framework builds a bridge between 3D cortical data and 2D models, making 2D pretrained models available for brain image analysis in cognitive and psychiatric neuroscience. 有限的样本量阻碍了深度学习在脑图像分析中的应用，迁移学习是一种可行的解决方案。然而，大多数预训练模型是基于2D 的，不能直接应用于3D 脑图像。在这项研究中，我们提出了一个新的框架，通过使用计算几何映射将基于表面的皮层形态测量投影到平面图像，将二维预训练模型应用于三维大脑图像。首先，使用 FreeSurfer 从磁共振成像(MRI)中重建三维皮层网格，然后基于保留区域的几何映射将其投影到具有拓扑保护的二维平面网格中。然后采用在 ImageNet 上预训练的二维深度模型进行形态测量学形状度量的皮层图像分类。我们将该框架应用于人类连接体项目数据集的性别分类，以及自闭症脑成像数据交换数据集的自闭症光谱分类(ASD)。此外，本文还提出了两阶段迁移学习策略，以性别分类作为中间任务来提高 ASD 分类性能。我们的框架通过迁移学习在性别分类和 ASD 分类方面带来了显著的改进。总之，提出的框架建立了三维皮层数据和二维模型之间的桥梁，使二维预训练模型可用于认知和精神神经科学的脑图像分析。 autism spectrum disorder 自闭症光谱, brain shape morphometry 脑形态测量学脑形态测量学, geometry mapping 几何映射几何映射, sex 做爱, transfer learning 转移学习 Issue Section: 发行组: Original Article 原文
Introduction简介Deep learning provides a powerful approach for brain imaging analysis (LeCun et al. 2015; Cole et al. 2017; Farooq et al. 2017; Kamnitsas et al. 2017; Shen et al. 2017; Mohsen et al. 2018). Deep convolutional neural network (CNN) (Simonyan and Zisserman 2014; He et al. 2016; Huang et al. 2017; Krizhevsky et al. 2017) has better representation capability and could automatically extract low-to-high-level spatial features (Gu et al. 2018; Abrol et al. 2021), usually outperforms conventional machine learning methods that need hand-crafted features. Inspired by the great success of deep learning in medical imaging analysis (Shen et al. 2017; Kermany et al. 2018; Sevakula et al. 2018; Ting et al. 2018; Zhu et al. 2020), neuroimaging researchers pay more attention to deep learning. Some studies use 3D CNN to analyze the 3D brain images since the brain is 3D (Valliani and Soni 2017; Oh et al. 2019; Bashyam et al. 2020; Thomas et al. 2020). The 3D models can preserve the spatial information of the brain and capture the features at the individual level. Nevertheless, they have too many parameters to be trained, leading to difficulty of converging. The computational efficiency is low due to the invalid calculation of volumes that contain null values. Moreover, the well-tuning of the models requires a large amount of data, but the collection of brain imaging data is expensive and time-consuming. 深度学习为脑成像分析提供了一种强有力的方法(LeVillage 等，2015; Cole 等，2017; Farooq 等，2017; Kamnitsas 等，2017; Shen 等，2017; Mohsen 等，2018)。深度卷积神经网络(CNN)(Simonyan and Zisserman 2014; He et al。2016; Huang et al。2017; Krizhevsky et al。2017)具有更好的表现能力，可以自动提取低层到高层的空间特征(Gu et al。2018; Abrol et al。2021) ，通常优于需要手工制作特征的传统机器学习方法。受到医学影像分析中深度学习的巨大成功的启发(Shen 等，2017; Kermany 等，2018; Sevakula 等，2018; Ding 等，2018; Zhu 等，2020) ，神经影像学研究人员更加关注深度学习。一些研究使用3D CNN 来分析3D 脑图像，因为大脑是3D 的(Valliani 和 Soni 2017; Oh 等，2019; Bashyam 等，2020; Thomas 等，2020)。三维模型可以保存大脑的空间信息，并在个体水平上捕捉特征。然而，它们有太多的参数需要训练，导致收敛困难。由于包含空值的体积计算无效，计算效率很低。此外，模型的良好调整需要大量的数据，但大脑成像数据的收集是昂贵的和耗时的。 In recent years, transfer learning has emerged as a crucial method for solving the insufficient training data problem by transferring knowledge from a source domain to a target domain (Pan and Yang 2009; Tan et al. 2018). The models are usually pretrained on a large-scale source dataset (e.g., ImageNet) (Deng et al. 2009) and then fine-tuned on the small target dataset in deep learning (Tan et al. 2018). The wealth of knowledge learned from the source dataset is implicitly encoded in huge parameters, making it possible for the target task to achieve better performance with limited samples. For brain imaging analysis, the surface-based cortical shape morphometry is closely related to sex, age, and neuropsychiatric disorders (Yuan et al. 2015; Bedford et al. 2020; Gharehgazlou et al. 2021). Many medical imaging studies use large-scale natural image datasets as the source domain and achieve better performance (Kermany et al. 2018; Sevakula et al. 2018; Ting et al. 2018; Zhu et al. 2020), indicating the feasibility of transfer learning from natural images to brain imaging data. However, most of the current pretrained models are designed for 2D planar images and cannot be directly applied to 3D brain magnetic resonance imaging (MRI). How to bridge the 3D brain images and 2D pretrained models remains unsolved. 近年来，通过将知识从源领域转移到目标领域，迁移学习已经成为解决训练数据不足问题的关键方法(Pan and Yang 2009; Tan et al。2018)。模型通常在大规模源数据集(例如 ImageNet)上进行预训练(Deng et al。2009) ，然后在深度学习中对小目标数据集进行微调(Tan et al。2018)。从源数据集中学到的大量知识隐式地编码在巨大的参数中，这使得目标任务可以在有限的样本下实现更好的性能。对于脑成像分析，基于表面的皮层形态测量与性别，年龄和神经精神障碍密切相关(Yuan 等，2015; Bedford 等，2020; Gharehgazlou 等，2021)。许多医学成像研究使用大规模的自然图像数据集作为源域，并获得更好的性能(Kermany 等，2018; Sevakula 等，2018; Ding 等，2018; Zhu 等，2020) ，表明将学习从自然图像转移到脑成像数据的可行性。然而，目前大多数预先训练的模型是为二维平面图像设计的，不能直接应用于三维脑磁共振成像(MRI)。如何将三维脑图像和二维预训练模型连接起来仍然是一个未解决的问题。 Most MRI studies based on CNN models use Euclidean distance as the metric for brain imaging analysis, ignoring the fact that the human brain has complex structures with folded sulcus and gyri (Fischl et al. 2008; Zhang et al. 2020). Nevertheless, the Euclidean distance is not the best metric of the brain images due to the non-Euclidean geometry of the complicated folding of the cerebral cortex (Seong et al. 2018). Treating brain images as ordinary images and applying Euclidean distance-based deep models directly to the brain images, which lack the neurobiological basis, may lead to the signal mixture of different brain regions and destruction of topological structure (Glasser et al. 2013). On the contrary, the distance along the brain surface is more consistent with neurobiology and cerebral cortex geometry (Fischl 2012; Glasser et al. 2013; Honnorat et al. 2015). For the aforementioned reasons, it is more reasonable to adopt the distance along the surface of the cerebral cortex instead of Euclidean distance in CNN models to obtain results with neurobiological significance. 大多数基于 CNN 模型的 MRI 研究使用欧几里得度量作为大脑成像分析的指标，忽略了人类大脑具有复杂结构和折叠的沟和回的事实(Fischl 等，2008; Zhang 等，2020)。尽管如此，由于大脑皮层复杂折叠的欧几里得度量非欧几里得几何(Seong et al。将脑图像作为普通图像处理，并将基于欧几里得距离的深度模型直接应用于缺乏神经生物学基础的脑图像，可能导致不同脑区的信号混合和拓扑结构的破坏(Glasser 等，2013)。相反，沿着脑表面的距离与神经生物学和大脑皮层几何形状更一致(Fischl 2012; Glasser 等，2013; Honnorat 等，2015)。基于上述原因，采用大脑皮层表面的距离代替 CNN 模型中的欧几里得度量来获得具有神经生物学意义的结果更为合理。 To address these issues, we propose a novel framework to bridge the gap between 3D MRI data and 2D CNN models by mapping the 3D cerebral cortex into 2D images and utilizing transfer learning to improve network performance. The proposed framework can be roughly divided into 4 steps. The first step is to process the cortical data with FreeSurfer and transform the cerebral cortex into 3D surface meshes and vertex-wise cortical shape metrics (Glasser et al. 2013). The 3D surface meshes are then topologically and equally mapped into 2D planar meshes through an area-preserving geometry mapping approach (Zhao et al. 2013) and further converted into 2D images for the subsequent analysis. The converted images reflect the distance along the brain surface of different brain regions, and the convolution on the converted images is similar to the convolution along the cortical surface, which is more consistent with neurological significance. The third step is to train the models using transfer learning. We choose the pretrained ResNet-50 (He et al. 2016) and DenseNet-121 (Huang et al. 2017) as the backbone networks. The pretrained models are fine-tuned with the acquired 2D images. Finally, the results from different metrics are ensembled using the stacking ensemble method (Wolpert 1992) to generate final individual-level classification results. The effectiveness of the proposed method is demonstrated with sex classification. 为了解决这些问题，我们提出了一种新的框架，通过将3D 大脑皮层映射成2D 图像，并利用传递学习来提高网络性能，从而在3D MRI 数据和2D CNN 模型之间架起一座桥梁。提出的框架大致可分为4个步骤。第一步是用 FreeSurfer 处理皮层数据，并将大脑皮层转换成3D 表面网格和顶点形状指标(Glasser et al。2013)。然后通过面积保持几何映射方法(Zhao et al。2013)将3D 表面网格拓扑和等量地映射到2D 平面网格中，并进一步转换成2D 图像以用于随后的分析。转换后的图像反映了不同脑区沿脑表面的距离，转换后图像的卷积与沿皮质表面的卷积相似，更符合神经学意义。第三步是利用迁移学习对模型进行训练。我们选择预先训练的 ResNet-50(He et al. 2016)和 DenseNet-121(Huang et al. 2017)作为骨干网络。预训练模型与获得的二维图像进行了微调。最后，使用叠加集成方法(Wolpert 1992)对不同度量的结果进行集成，生成最终的个体层次分类结果。性别分类验证了该方法的有效性。 Moreover, previous studies have reported sex differences of structures and functions in autism spectrum disorder (ASD) (Bejerot et al. 2012; Loomes et al. 2017; Bedford et al. 2020; Liu et al. 2020). A reasonable assumption is that the sex-related features may be helpful in ASD classification. Thus, we further develop a 2-stage transfer learning framework for the classification of ASD by using the sex classification of healthy people as an intermediate task to reduce the distribution differences of the source domain and the target domain for better performance. 此外，先前的研究报道了自闭症光谱结构和功能的性别差异(Bejerot 等，2012; Loomes 等，2017; Bedford 等，2020; Liu 等，2020)。一个合理的假设是，性别相关特征可能有助于 ASD 的分类。因此，我们进一步发展了一个两阶段的迁移学习框架，以健康人的性别分类为中间任务，以减少源域和目标域的分布差异，以获得更好的表现。 The contributions of this paper can be summarized as follows: 本文件的贡献可概述如下:
Materials and Methods材料及方法Data and Preprocessing数据和预处理The data used in sex classification come from the Human Connectome Project (HCP) S1200 release (Van Essen et al. 2012), including 1113 subjects (505 females vs. 606 males). Subjects are scanned in 3 T Siemens scanners in Washington University with the following parameters: spatial resolution = 2 × 2 × 2 mm3, time repetition (TR) = 720 ms, time echo (TE) = 33.1 ms, field of view (FoV) = 208 × 180 mm2, slices = 72, flip angle = 52 degrees. Male and female subjects are matched in age and education. 性别分类中使用的数据来自人类连接组项目(HCP) S1200发布(Van Essen 等，2012) ，包括1113名受试者(505名女性对606名男性)。在华盛顿大学的3T 西门子扫描仪中扫描受试者，参数如下: 空间分辨率 = 2 × 2 × 2mm3，时间重复(TR) = 720ms，时间回波(TE) = 33.1 ms，视场(FoV) = 208 × 180mm2，切片 = 72，翻转角 = 52度。男性和女性受试者在年龄和教育程度上是匹配的。 We use the large-scale publicly available dataset, the Autism Brain Imaging Data Exchange (ABIDE) dataset (Di Martino et al. 2014; Di Martino et al. 2017), for ASD classification. The ABIDE dataset consists of 2 subsets: ABIDE I and ABIDE II. ABIDE I contains 1112 subjects (539 ASD patients vs. 573 normal controls) collected from 16 sites and ABIDE II comprises 1114 subjects (521 ASD patients vs. 593 normal controls) collected from 19 sites. We first discard 219 samples from 2226 samples of ABIDE dataset whose scans fail to complete all steps of the Freesurfer preprocessing pipeline due to low-image quality (Backhausen et al. 2016). In addition, Freesurfer sometimes generates incorrect segmentation owing to the low-image quality and the challenging of whole-brain reconstruction, even though the sample passes the pipeline. So the segmentation quality is further checked by visual inspection, and 13 subjects whose segmentations are incorrect are excluded. Finally, a total of 1994 subjects are involved in the following analysis. 我们使用大规模公开可用的数据集，自闭症脑成像数据交换(ABIDE)数据集(Di Martino 等，2014; Di Martino 等，2017)进行 ASD 分类。ABIDE 数据集由两个子集组成: ABIDE I 和 ABIDE II。ABIDE I 包含从16个站点收集的1112个受试者(539个 ASD 患者对573个正常对照) ，ABIDE II 包含从19个站点收集的1114个受试者(521个 ASD 患者对593个正常对照)。我们首先丢弃2226个 ABIDE 数据集样本中的219个样本，由于图像质量低，其扫描未能完成 Freesurfer 预处理流水线的所有步骤(Backhausen 等，2016)。此外，Freesurfer 有时会产生不正确的分割，由于低图像质量和全脑重建的挑战，即使样本通过管道。进一步通过视觉检测对分割质量进行了检验，排除了13个分割不正确的被试。最后，对1994名研究对象进行了以下分析。 As with many other MRI studies, we focus on the cerebral cortex. The cerebral cortex can be regarded as a thin folding surface with heterogeneous thickness, so it is impossible to transform it into a 2D image directly. In this study, the structural MRI preprocessing pipeline of FreeSurfer is adopted to preprocess data from both HCP and ABIDE (Glasser et al. 2013). The pipeline includes the segmentation of T1w volume, tessellation and topology correction of the initial white matter surface, spherical inflation of the white matter surface, registration to the fsaverage surface template, segmentation of sulci and gyri, pial surface generation, surface and volume anatomical parcellations, and morphometric measurements (Fischl 2012; Glasser et al. 2013). The 32 k cortical meshes and vertex-wise cortical shape metrics, including thickness, sulcal depth, curvature, and myelin map, are generated from the cerebral cortex. Due to the lack of T2-weighted images, the myelin map for ABIDE is unavailable. 和其他许多核磁共振成像研究一样，我们关注的是大脑皮层。大脑皮层可以看作是一个具有不均匀厚度的薄折叠表面，因此不可能将其直接转换成二维图像。在这项研究中，FreeSurfer 的结构 MRI 预处理流水线被用来预处理来自 HCP 和 ABIDE 的数据(Glasser et al。2013)。管道包括 T1w 体积的分割，初始白质表面的镶嵌和拓扑校正，白质表面的球形膨胀，对平均表面模板的配准，脑沟和脑回的分割，软脑膜表面生成，表面和体积解剖分区以及形态测量(Fischl 2012; Glasser 等，2013)。32k 的皮质网格和顶点状的皮质形状度量，包括厚度，沟深，曲率和髓鞘图，是由大脑皮质产生的。由于缺乏 T2加权像，ABIDE 的髓鞘图不可用。 Geometry Mapping几何映射As mentioned above, each metric is composed of surfaces from 2 hemispheres. To adapt the 3D imaging data to 2D models, we need to map 3D cortical meshes into 2D images. A 3D mesh generated by FreeSurfer is a folded closed surface that could not be directly mapped into a planar mesh. However, vertexes corresponding to the medial wall that is close to the subcortical regions would have null values. We remove these vertexes and thus obtain unclosed meshes, which can be theoretically mapped into a regular planar mesh using geometry mapping approaches. Conformal mapping (Wang et al. 2011) and area-preserving mapping (Su et al. 2013; Zhao et al. 2013) are 2 commonly used geometry mapping approaches that map irregular 3D meshes as regular planar meshes. The former keeps the mapping of angles but leads to area distortion, and the latter will cause the opposite effects (Nadeem et al. 2016). The area distortion may seriously influence the training of deep models, so we adopt the area-preserving mapping approach. Considering the compatibility with deep learning, we map the brain image into a unit rectangle. 如上所述，每个度量由来自两个半球的表面组成。为了使三维成像数据适应二维模型，我们需要将三维皮层网格映射成二维图像。由 FreeSurfer 生成的3D 网格是一个不能直接映射到平面网格的折叠封闭表面。然而，对应于靠近皮质下区域的内侧壁的顶点将具有空值。我们删除这些顶点，从而获得非封闭网格，这可以在理论上映射到一个规则的平面网格使用几何映射方法。保形映射(Wang et al。2011)和面积保持映射(Su et al。2013; Zhao et al。2013)是两种常用的几何映射方法，将不规则的3D 网格映射为正则的平面网格。前者保持角度映射，但导致面积失真，后者将导致相反的效果(Nadeem et al。2016)。面积畸变会严重影响深度模型的训练，因此采用了面积保持映射方法。考虑到与深度学习的兼容性，我们将大脑图像映射成一个单位矩形。 Supposing 假设(S,g) is the input surface mesh in 是输入曲面的网格R3 with Riemannian metric 用黎曼度量g, there is a unique conformal mapping 有一个唯一的保角映射∅:S→D according to the Riemann mapping theorem, where 根据黎曼映射定理D is a unit square with 4 corners predefined as 4 vertexes equally distributed along the surface edge. Then there is a unique Brenier mapping 是一个单位正方形与4个角预定义为4个顶点均匀分布沿表面边缘。然后是一个独特的布雷尼尔映射τ:(D,dxdy)→(D,μ), which makes sure the area of each cell is preserved. The area-preserving mapping ( ，以确保每个单元格的面积得到保留Zhao et al. 2013 Zhao 等人2013年) is the combination of the Riemann mapping and Brenier mapping: )是黎曼映射和布雷尼尔映射的结合:τ−1∘∅:S→D. In practice, the conformal mapping procedure can be implemented with the discrete Ricci Flow method. Supposing the vertexes are 在实际应用中，可以用离散 Ricci 流方法实现保角映射过程V={v1,v2,...,vn}, the curvature is 曲率是K, and the target curvature is 目标曲率是K¯¯¯¯¯, the conformal factor is defined as 保形因子定义为u:V→R, and the discrete Ricci flow can be represented by ( 离散的 Ricci 流可以用(Wang et al. 2011 Wang 等人2011年; Zhao et al. 2013 Zhao 等人2013年) du(vi)dt=K¯¯¯¯¯(vi)−K(vi,t) The unique solution is the optimal point of the discrete flow energy 唯一解是离散流动能量的最优点 E(u)=∫u0∑i=1n(K¯¯¯¯¯(vi)−K(vi))dui, Then the conformal mapping needs to be converted into area-preserving mapping using optimal mass transport to make the areas of cells 然后利用最优质量传输将保角映射转化为面积保持映射，得到单元面积w={w1,w2,...,wn} close to original areas 接近原来的区域w¯¯¯¯={w¯¯¯¯1,w¯¯¯¯2,...,w¯¯¯¯n}, where ，在哪里w¯¯¯¯i represent the initial area of 表示... 的初始面积vi. Assigning the height vector of the cells as 。将单元格的高度向量指定为h={h1,h2,…,hn}. For any given measurement 。任何给定的测量w¯¯¯¯, there must exist a unique ，必须存在一个独特的h that satisfies the area-preservation constraints for all cells. The Brenier mapping can be calculated with the energy function 满足所有单元的面积保持约束。布雷尼尔映射可以用能量函数来计算 E¯¯¯¯(h)=∑i=1nw¯¯¯¯ihi−∫h0∑j=1nwjdhj, This study focuses on the images of brain shape metrics, but the features generated in the surface parameterization process, such as Beltrami coefficients, also contain discriminative information (Su et al. 2013). We set the target curvature to π/2 for 4 corners and 0 for other vertexes to generate a rectangular mesh. The 3D cortical mesh is then topologically and equably mapped onto a rectangular planar mesh without any tearing or overlaps after the geometry mapping. The acquired planar mesh cannot be directly utilized by convolutional networks, and the brain images need to have the same data form as the natural images used in the pretrained models. So it is necessary to transform it into 2D images. We use a weighted triangular interpolation approach based on barycentric coordinates to avoid the null values in the 2D images. The average of the points that fall within the same pixel is taken as the pixel’s value. 这项研究的重点是大脑形状指标的图像，但在表面参量化过程中产生的特征，如贝尔特拉米系数，也包含有区分信息(Su et al。2013)。对于4个角，我们将目标曲率设置为 π/2，对于其他顶点，我们将目标曲率设置为0，以生成一个矩形网格。然后，在几何映射之后，将三维皮层网格拓扑和均匀地映射到一个矩形平面网格上，没有任何撕裂或重叠。所获得的平面网格不能直接用于卷积网络，而且脑图像需要与预训练模型中使用的自然图像具有相同的数据形式。因此，有必要将其转换成二维图像。我们使用一个基于质心座标的加权三角插值方法来避免二维图像中的零值。落在同一像素内的点的平均值作为该像素的值。 Training with CNN和 CNN 一起训练Deep Model Architecture深度模型体系结构After the area-preserving geometry mapping, the vertex-wise cortical shape metrics are mapped as 224 × 224 images. To demonstrate the reliability of our method, we use 2 popular deep convolutional networks (i.e., ResNet-50 and DenseNet-121) for experiments. ResNet adds skip connections between the adjacent layers and calculates residuals from inputs to outputs. It alleviates the gradient disappearance in deep learning and achieves better performance. DenseNet introduces skip connections between every 2 layers and uses concatenation operation instead of summation operation used in ResNet. Both models have been demonstrated to be robust and efficient in image classification, and the corresponding pretrained models are widely used and available online. 经过面积保持的几何映射，顶点方向的皮层形状度量映射为224 × 224图像。为了证明该方法的可靠性，我们使用了2个流行的深卷积网络(即 ResNet-50和 DenseNet-121)进行实验。ResNet 添加了相邻层之间的跳跃连接，并计算从输入到输出的残差。它缓解了深度学习中的梯度消失现象，取得了较好的效果。DenseNet 在每2层之间引入跳跃连接，并使用串联操作代替 ResNet 中使用的求和操作。两种模型在图像分类中均具有较强的鲁棒性和较高的分类效率，相应的预训练模型得到了广泛的应用和在线应用。 Transfer Learning from ImageNet基于 ImageNet 的迁移学习Transfer learning is utilized to improve network performance on small datasets. Deep neural networks are first pretrained on the large-scale natural image dataset ImageNet. Then the fully connected layer is replaced to meet the class number of the target task, and the pretrained models are fine-tuned using the acquired 2D images of different metrics, respectively. The 10-fold cross-validation strategy is used to test the reliability of the classification performance. The samples are randomly shuffled and divided into 10-folds, without consideration of scanning site, sex, and patient/control ratio. In each experiment, we use 9-folds for training and the left fold for testing. Before training, the input images I are normalized, and the resulted images I^ can be formulated as I^=(I−μ)/σ, where μ and σ are the mean value and standard deviation of the input images, respectively. 利用传递学习提高小数据集上的网络性能。首先在大规模自然图像数据集 ImageNet 上对深度神经网络进行预训练。然后替换完全连通的层以满足目标任务的类数，并分别利用获得的不同度量的二维图像对预训练后的模型进行微调。10倍交叉验证策略用于测试分类性能的可靠性。将样本随机分为10组，不考虑扫描部位、性别和患者/对照组比例。在每个实验中，我们使用9倍的训练和测试左折叠。在训练之前，输入图像 I 被标准化，得到的图像 I^ 可以表示为 I^=(I−μ)/σ，其中 μ 和 σ 分别是输入图像的平均值和标准差。 Mix-Up混乱A mix-up strategy is also introduced in the training procedure ( 在训练过程中还引入了混合策略(Zhang et al. 2018 Zhang 等人2018年). Mix-up is a widely used data augmentation method in computer vision. It uses the linear interpolation of 2 random samples and their labels as virtual samples to improve the generalization capability of the network. The mix-up can be formulated as xmix=λxi+(1−λ)xjymix=λyi+(1−λ)yj, Ensemble Using Stacking使用堆叠的合奏After the training and testing procedure, we obtain the results of different metrics of both hemispheres. It is necessary to ensemble the metric-level results to generate the final individual-level classification results. Instead of simple voting or weighted voting methods, we adopt the hierarchical model ensemble method, that is, stacking, for individual-level ensemble (Wolpert 1992). Specifically, the results of different metrics are concatenated as the new input features of the individual-level classification model. The extreme gradient boosting (XGboost) (Chen and Guestrin 2016) is adopted as the stacking model. Compared with voting-based ensemble methods, stacking can automatically learn the weights of the input features and usually gets better results. The hyperparameters of XGboost are optimized with the grid search method. 经过训练和测试过程，我们得到了两个半球不同度量的结果。为了生成最终的个体层次分类结果，有必要对度量层次结果进行集成。我们不采用简单的投票或加权投票的方法，而是采用层次模型集成的方法，即叠加，为个人水平集成(Wolpert 1992)。具体来说，不同度量的结果被连接起来作为个人层次分类模型的新输入特征。采用极端梯度提升(xgost)(Chen and Guestrin 2016)作为堆叠模型。与基于投票的集成方法相比，叠加方法能够自动学习输入特征的权重，通常能够得到更好的结果。利用网格搜索方法对 XGBoost 的超参数进行了优化。 Two-Stage Transfer Learning两阶段迁移学习For the classification of ASD, we introduce the 2-stage transfer learning approach for better performance. Although it is popular to use the models pretrained with large-scale natural image datasets on other fields, the domain differences between the source and target datasets will still affect the effectiveness of transfer learning (Jean et al. 2016). An effective approach is to use an intermediate domain to bridge the source domain and target domain. The model is first transferred from the source domain to the intermediate domain and then transferred from the intermediate domain to the target domain. For neuropsychiatric disorders such as ASD, sex classification of healthy people is an excellent intermediate domain task. Firstly, brain images from healthy people and ASD patients have similar features. Compared with neuropsychiatric disorders, healthy people’s data usually have better homogeneity. Moreover, sex labels are credible, which is vital in brain imaging analysis. In the 2-stage transfer learning, we first convert both the intermediate domain (HCP) and the target domain (ABIDE) from 3D MRI data to 2D images with the Freesurfer pipeline. The models are transferred from ImageNet to the sex classification of healthy people with the HCP dataset. Then the acquired models are further transferred to the classification of ASD with the ABIDE dataset. In the 2-stage transfer learning framework, the models are fine-tuned twice using the intermediate domain and the target domain, respectively. 对于 ASD 的分类，我们引入了两阶段迁移学习方法以获得更好的性能。尽管在其他领域使用大规模自然图像数据集预训练的模型是流行的，但是源和目标数据集之间的领域差异仍然会影响转移学习的有效性(Jean et al。2016)。一种有效的方法是使用一个中间域来连接源域和目标域。模型首先从源域转移到中间域，然后从中间域转移到目标域。对于像 ASD 这样的神经精神障碍，健康人的性别分类是一个很好的中间领域的任务。首先，健康人和自闭症患者的大脑图像具有相似的特征。与神经精神障碍相比，健康人的数据通常具有更好的同质性。此外，性别标签是可信的，这是至关重要的脑成像分析。在两阶段传输学习中，我们首先使用 Freesurfer 流水线将三维 MRI 数据中的中间域(HCP)和目标域(ABIDE)转换为二维图像。模型通过 HCP 数据集从 ImageNet 转移到健康人的性别分类。然后利用 ABIDE 数据集将获得的模型进一步转化为 ASD 的分类。在两阶段迁移学习框架中，分别使用中间域和目标域对模型进行两次微调。 Visualization想象To interpret the classification results of the models and locate the cortical shape morphometric differences, the occlusion test is adopted to measure the importance of different regions in the classification (Zeiler and Fergus 2014). Specifically, we cover the image with a 30 × 30 black square and calculate the accuracy drop, which is regarded as the importance of the covered region in the classification. Then we move the square to the next region with a stride of 4 until the whole image is covered. We finally get occlusion test maps and resize them to the same size as the original image. We use the average of all images as the final results. The occlusion test results are then reconstructed as 3D meshes and visualized with Connectome Workbench visualization software. 为了解释模型的分类结果和定位皮层形态学差异，采用遮挡测试来测量不同区域在分类中的重要性(Zeiler and Fergus 2014)。具体地说，我们用一个30 × 30的黑色正方形覆盖图像，并计算精度下降，这被认为是覆盖区域在分类中的重要性。然后我们将正方形移动到下一个区域，步长为4，直到整个图像被覆盖。我们最终得到遮挡测试图，并将它们调整到与原始图像相同的大小。我们使用所有图像的平均值作为最终结果。然后将遮挡测试结果重建为三维网格，并用 Connectome Workbench 可视化软件进行可视化。 Results结果Training Details培训详情The models are trained on an Ubuntu 18.04.1 server with 2 8-core Intel E5 2609 1.7 GHz processors and 4 NVIDIA GTX-V100 graphical processing units. The code is written in Python and Pytorch framework (Paszke et al. 2019). The models pretrained on ImageNet are acquired from torchvision (https://download.pytorch.org/models/). Each model is trained for 125 epochs with a batch size of 64, of which the first 75 epochs are trained with mix-up, and the latter 50 epochs are trained with original data, and the model of the last epoch is retained for testing. Stochastic gradient descent and cross-entropy loss are adopted for model optimization. The learning rate is set to 0.01 initially, divided by 10 every 25 epochs for the first 75 epochs, and is then fixed at 0.0001 for the following 50 epochs. The momentum is set to 0.9. The hyperparameters are optimized using the grid search strategy. 这些机型在 Ubuntu 18.04.1服务器上进行培训，该服务器配有2个8核 Intel E526091.7 GHz 处理器和4个 NVIDIA GTX-V100图形处理单元。代码是用 Python 和 Pytorch 框架编写的(Paszke 等，2019)。在 ImageNet 上预先训练的模型是从 torchvision ( https://download.pytorch.org/models/)获得的。每个模型训练125个历元，批量大小为64个，其中前75个历元用混合训练，后50个历元用原始数据训练，最后一个历元的模型保留用于测试。模型优化采用随机梯度下降和交叉熵损失。最初将学习率设置为0.01，除以前75个纪元中每25个纪元的10个，然后将后50个纪元的学习率固定为0.0001。动量设定为0.9。利用网格搜索策略对超参数进行了优化。 Area-Preserving Geometry Mapping Results面积保持几何映射结果The sketch diagram of area-preserving geometry mapping for cortical meshes is shown in Figure 1. To visually compare the 3D cortical meshes and the corresponding 2D images, we map the Desikan–Killiany (D–K) atlas (Desikan et al. 2006) as a 2D planar atlas (Fig. 2). Brain regions are illustrated in different colors for visualization. For the 3D atlas, a series of different views are needed to show the complete information of the whole brain, and some regions are still hard to observe due to the complex folding of the cerebral cortex. However, our 2D atlas can avoid these disadvantages and show the whole brain without occlusion in one view, which demonstrates the potential and superiority of our method in the visualization of brain images. 皮层网格的面积保持几何映射示意图如图1所示。为了在视觉上比较3D 皮层网格和相应的2D 图像，我们将 Desikan-Killiany (D-K)图谱(Desikan 等，2006)映射为2D 平面图谱(图2)。大脑区域以不同的颜色显示。对于三维地图集，需要一系列不同的视图来显示整个大脑的完整信息，而且由于大脑皮层的复杂折叠，一些区域仍然难以观察到。然而，我们的二维地图集可以避免这些缺点，并显示整个大脑没有遮挡在一个视图，这表明我们的方法在可视化的脑图像的潜力和优越性。 Figure 1 图1 Overview of the proposed framework. FreeSurfer is used to generate 3D cortical meshes and vertex-wise cortical shape metrics. The 3D mesh is then converted into a planar mesh using area-preserving geometry mapping. Different metrics are calculated using transfer learning with ImageNet, and then the results are further ensembled with a stacking approach to get individual-level results. The sketch diagram of the geometry mapping is also displayed. The red points distributed evenly on the edge are the selected corners. 拟议架构概览。FreeSurfer 用于生成3D 皮层网格和顶点形状度量。然后使用面积保持几何映射将三维网格转换成平面网格。使用 ImageNet 的转移学习计算不同的度量，然后将结果进一步集成到一个堆叠方法中，以获得个体级别的结果。还显示了几何映射的示意图。在边缘上均匀分布的红点是被选中的角。 Figure 2 图2 The 3D D-K atlas (A) with different views and the 2D D–K atlas (B) generated by our method. The corresponding brain regions of 2 atlases are shown in the same colors. 该方法生成了具有不同视图的三维 D-K 图谱(A)和具有不同视图的二维 D-K 图谱(B)。2个地图集的相应脑区显示为相同的颜色。 Sex Classification Results性别分类结果Sex classification is a fundamental problem in brain imaging analysis. There is a long debate about whether male and female brains are distinguishable, and many studies attempt to solve the problem with machine learning methods (Weis et al. 2020). We perform the sex classification task on the HCP dataset, and the results are shown in Table 1. Two different deep models are adopted to measure the effectiveness of the proposed method. For comparison, we test the models trained from scratch first and achieve 89.67% accuracy for ResNet and 92.99% accuracy for DenseNet. Furthermore, we test transfer learning by transferring the models pretrained on the source dataset (ImageNet) to the target dataset (HCP). Transfer learning achieves an accuracy of 94.34% for ResNet and 95.06% for DenseNet, resulting in improvements of 4.67% and 2.07% in accuracy. The results demonstrate that transfer learning could significantly boost classification performance. The receiver operating characteristic (ROC) curves and confusion matrices are shown in Supplementary Figures S1 and S2, respectively. The proposed method achieves the best area under ROC curves (AUC) score of 0.9854. The high accuracy of our experiment on sex classification demonstrates the effectiveness of our framework and suggests that males and females are distinguishable with the cortical shape metrics revealed by structural MRI. 性别分类是脑成像分析中的一个基本问题。关于男性和女性的大脑是否可以区分，有一个长期的争论，许多研究试图用机器学习方法来解决这个问题(Weis et al. 2020)。我们在 HCP 数据集上执行性别分类任务，结果如表1所示。采用两种不同的深度模型来衡量该方法的有效性。为了进行比较，我们首先测试了从头开始训练的模型，ResNet 的准确率为89.67% ，DenseNet 的准确率为92.99% 。此外，我们通过将源数据集(ImageNet)上预先训练的模型转移到目标数据集(HCP)来测试迁移学习。转移学习在 ResNet 和 DenseNet 中分别达到了94.34% 和95.06% 的准确率，分别提高了4.67% 和2.07% 的准确率。结果表明，迁移学习可以显著提高分类性能。ROC曲线(ROC)曲线和混淆矩阵分别显示在补充图 S1和 S2中。该方法在 ROC 曲线下的最佳面积为0.9854。我们的性别分类实验的高准确性证明了我们的框架的有效性，并表明男性和女性是可以区分的皮质形状指标由结构 MRI 显示。 Table 1 表1 Sex classification results of cerebral cortex based on HCP
Acc, Sen, and Spc refer to accuracy, sensitivity, and specificity, respectively. The transfer refers to transfer learning from ImageNet to HCP. The best accuracy, sensitivity, specificity, and AUC are shown in bold. Table 1 Sex classification results of cerebral cortex based on HCP 基于 HCP 的大脑皮层性别分类结果
Acc, Sen, and Spc refer to accuracy, sensitivity, and specificity, respectively. The transfer refers to transfer learning from ImageNet to HCP. The best accuracy, sensitivity, specificity, and AUC are shown in bold. Acc、 Sen 和 Spc 分别指准确性、敏感性和特异性。转移是指从 ImageNet 到 HCP 的转移学习。最佳的准确性、灵敏度、特异性和 AUC 以粗体显示。 The results of different metrics are also calculated with the stacking method (Fig. 3). Results from 2 hemispheres are concatenated as the inputs of the individual-level classifiers. The metrics-level accuracies are improved with a range of 4.49–11.01% and 1.53–4.13% for ResNet and DenseNet with transfer learning, respectively. The significant improvement with transfer learning on each metric demonstrates the stability and effectiveness of our framework. The myelin map achieves the best accuracy. The curvature performs worse than other metrics but gains the most significant improvement with transfer learning. 不同指标的计算结果也采用了叠加法(图3)。来自两个半球的结果被串联起来作为个体层次分类器的输入。通过转移学习，ResNet 和 DenseNet 的度量级精度分别提高了4.49-11.01% 和1.53-4.13% 。在每个指标上的迁移学习的显著改进证明了我们框架的稳定性和有效性。髓鞘图达到了最佳的准确性。曲率比其他度量表现更差，但是通过迁移学习获得了最显著的改善。 Figure 3 图3 Classification results of single metric on sex classification (left) and ASD classification (right). In sex classification, the results on thickness, sulcal depth, curvature, and myelin map are shown to investigate the effectiveness of transfer learning under our framework. In ASD classification, the results of thickness, sulcal depth, and curvature are shown to observe performance improvement with the proposed method. The “transfer” represents the transfer learning from ImageNet to the target dataset, and the “2 stage” refers to the transfer learning from ImageNet to ABIDE with HCP as the intermediate domain. 性别分类单指标分类结果(左)和 ASD 分类单指标分类结果(右)。在性别分类中，结果显示厚度，沟深度，曲率和髓鞘地图，以调查有效的迁移学习在我们的框架下。在 ASD 分类中，通过对厚度、沟深和曲率的分析，可以观察到该方法对 ASD 分类性能的改善。“转移”是指从 ImageNet 到目标数据集的转移学习，“2阶段”是指以 HCP 为中间域的从 ImageNet 到 ABIDE 的转移学习。 Moreover, we explore the effects of total intracranial volume (TIV) on sex classification (Sanchis-Segura et al. 2020) under our framework. The results show that our framework still works well with matched TIV (Supplementary Table S1). 此外，在我们的框架下，我们探讨了颅内总体积(TIV)对性别分类的影响(Sanchis-Segura 等，2020)。结果表明，我们的框架仍然与匹配的 TIV 工作得很好(补充表 S1)。 ASD Classification ResultsASD 分类结果We further apply our method to a multisite ASD dataset (ABIDE) to distinguish patients from healthy controls. Due to the lack of T2-weighted images, the myelin maps are not available for the ABIDE dataset, so we only use thickness, sulcal depth, and curvature for the classification of ASD. The results are shown in Table 2. 我们进一步将我们的方法应用于多位点 ASD 数据集(ABIDE) ，以区分患者和健康对照。由于缺乏 T2加权像，ABIDE 数据集无法获得髓鞘图，因此我们只能使用厚度、沟深和曲率对 ASD 进行分类。结果如表2所示。 Table 2 表2 ASD classification results of cerebral cortex on ABIDE
The transfer refers to direct transfer learning from ImageNet to ABIDE, whereas the 2 stage represents the ImageNet-HCP-ABIDE transfer learning strategy. The PCA + SVM, DenseNet (slice), and DenseNet (3D volume) are based on the 3D cerebral cortex for a fair comparison. The DenseNet (3D mesh) is based on the 3D cortical shape metrics. The best accuracy, sensitivity, specificity, and AUC are shown in bold. Table 2 表2 ASD classification results of cerebral cortex on ABIDE 大脑皮层 ASD 在 ABIDE 上的分类结果
The transfer refers to direct transfer learning from ImageNet to ABIDE, whereas the 2 stage represents the ImageNet-HCP-ABIDE transfer learning strategy. The PCA + SVM, DenseNet (slice), and DenseNet (3D volume) are based on the 3D cerebral cortex for a fair comparison. The DenseNet (3D mesh) is based on the 3D cortical shape metrics. The best accuracy, sensitivity, specificity, and AUC are shown in bold. 迁移是指从 ImageNet 到 ABIDE 的直接迁移学习，而两个阶段代表 ImageNet-HCP-ABIDE 迁移学习策略。PCA + SVM，DenseNet (切片)和 DenseNet (3D 体积)是基于3D 大脑皮层进行公平比较。致密网(3D 网格)是基于3D 皮质形状度量。最佳的准确性、灵敏度、特异性和 AUC 以粗体显示。 We first train and test ResNet and DenseNet from scratch and obtain the accuracies of 63.04% and 63.64%, respectively. Furthermore, we test direct transfer learning from ImageNet to the ABIDE dataset and achieve the accuracies of 65.89% for ResNet and 65.59% for DenseNet. Then the 2-stage transfer learning is tested based on the hypothesis that sex classification on healthy people can provide valuable features for the diagnostic classification of neuropsychiatric disorders. The models are transferred from ImageNet to HCP first and further transferred to ABIDE. The 2-stage transferred ResNet and DenseNet achieve the accuracies of 67.70% and 67.85%, respectively. The direct transfer learning from ImageNet brings increases of 2.85% for ResNet and 1.95% for DenseNet. Although the 2-stage transfer learning achieves improvements of 4.66% for ResNet and 4.21% for DenseNet in accuracy, the best AUC score is 0.7237. 首先从零开始对 ResNet 和 DenseNet 进行训练和测试，得到的准确率分别为63.04% 和63.64% 。此外，我们测试了从 ImageNet 到 ABIDE 数据集的直接转移学习，结果表明，对于 ResNet 和 DenseNet，准确率分别为65.89% 和65.59% 。然后基于健康人的性别分类可以为神经精神障碍的诊断分类提供有价值的特征的假设，对两阶段迁移学习进行了检验。模型首先从 ImageNet 转移到 HCP，然后再转移到 ABIDE。两级传输的 ResNet 和 DenseNet 的准确率分别为67.70% 和67.85% 。ImageNet 的直接转移学习使 ResNet 增加了2.85% ，DenseNet 增加了1.95% 。虽然两阶段迁移学习在准确性方面对 ResNet 和 DenseNet 分别提高了4.66% 和4.21% ，但最佳 AUC 评分为0.7237。 Moreover, we validate transfer learning from HCP to ABIDE to investigate the role of sex classification in 2-stage transfer learning. ResNet and DenseNet achieve accuracies of 65.44% and 65.25%, respectively, indicating that the pretraining on sex classification is helpful for the ASD classification. 此外，我们验证了从 HCP 到 ABIDE 的迁移学习，以探讨性别分类在两阶段迁移学习中的作用。ResNet 和 DenseNet 分别达到了65.44% 和65.25% 的准确率，说明性别分类的预训练有助于 ASD 分类。 The results of thickness, sulcal depth, and curvature are shown in Figure 3. The results of single metrics are consistent with the individual-level results. Direct transfer learning achieves better results than training from scratch, whereas the 2-stage transfer learning reaches the highest accuracies in all metrics. Thickness seems to perform better in the ASD classification, whereas sulcal depth and curvature achieve considerable performance. Compared with direct transfer learning, 2-stage transfer learning brings more performance improvements in thickness and sulcal depth. 厚度、沟深和曲率的结果如图3所示。单个指标的结果与个人层面的结果是一致的。直接迁移学习比从头开始的学习效果更好，而两阶段迁移学习在所有指标中都达到了最高的精度。厚度似乎表现更好的 ASD 分类，而沟深度和曲率取得了相当大的性能。与直接迁移学习相比，两阶段迁移学习在厚度和沟深方面有更大的提高。 Moreover, we use the leave-one-site-out cross-validation to investigate the performance of different models on unseen sites, which can further demonstrate the generalization ability. In each experiment, one site is used as the testing set, and the rest are used as the training set. The results are shown in Supplementary Table S2. The model trained from scratch, transfer learning, and 2-stage transfer learning achieve lower accuracies (61.34%, 63.45%, 65.41%) than those in 10-fold cross-validation. It is reasonable because testing on the unseen site is usually more difficult. However, the classification performance benefits from transfer learning and 2-stage transfer learning in leave-one-site-out cross-validation as well, indicating the robustness and effectiveness of the proposed framework. 此外，我们使用剩余一个站点的交叉验证来研究不同模型在未见站点上的表现，这可以进一步证明推广能力。在每个实验中，一个站点作为测试集，其余站点作为训练集。结果见补充表 S2。从头开始训练的模型、迁移学习和2阶段迁移学习的准确率(61.34% 、63.45% 、65.41%)低于10倍交叉验证的模型。这是合理的，因为在看不见的站点上进行测试通常更加困难。然而，在「留一个地点」的交叉验证中，迁移学习和两阶段迁移学习均有助分类表现，显示建议架构的稳健性和成效。 Comparison With Other Methods on ASD ClassificationASD 分类方法与其他方法的比较Many methods have been used for ASD classification based on the ABIDE dataset (Sabuncu et al. 2015; Aghdam et al. 2018; Monté-Rubio et al. 2018; Arya et al. 2020; Shahamat and Abadeh 2020). However, the sample size and brain features used in these studies vary a lot, making it difficult for horizontal comparison. To better measure the property of the proposed method and make a fair comparison, we compare the proposed framework with 4 other methods, including the support vector machine (SVM) (Chang and Lin 2011), slice-based 2D CNN, volume-based 3D CNN, and mesh-based 3D CNN using identical samples and brain features. Since ResNet and DenseNet have comparable performance, we only test the corresponding methods using the DenseNet architecture. We only consider the cerebral cortex in these experiments for a fair comparison. 根据 ABIDE 数据集(Sabuncu 等，2015; Aghdam 等，2018; Monté-ubio 等，2018; Arya 等，2020; Shahamat 和 Abadeh 2020) ，已经使用了许多方法进行 ASD 分类。然而，在这些研究中使用的样本量和大脑特征差异很大，使得横向比较很困难。为了更好地测量提出的方法的特性并进行公平的比较，我们将提出的框架与其他4种方法进行了比较，包括使用相同样本和大脑特征的支持向量机(SVM)(Chang and Lin 2011)、基于切片的2D CNN、基于体积的3D CNN 和基于网格的3D CNN。由于 ResNet 和 DenseNet 具有相当的性能，我们只使用 DenseNet 体系结构测试相应的方法。在这些实验中，我们只考虑大脑皮层，以便进行公平的比较。 The results of these methods are shown in Table 2. SVM is one of the classic methods for brain MRI. The data are preprocessed and reshaped into a vector, and the principal component analysis (PCA) (Wold et al. 1987) is adopted for feature extraction. An accuracy of 58.12% is achieved with SVM, which is significantly lower than that of the proposed 2-stage transfer learning, indicating the superiority of our framework. Some studies use 2D CNNs to analyze brain images by cutting them into slices. Similarly, in the slice-based DenseNet, the 3D brain images are cut into slices in 3 directions, and the pretrained models are fine-tuned using the acquired slices. The results of the slices are finally ensembled with stacking. The mix-up, stacking, and transfer learning strategies are adopted to ensure a fair comparison. Although the brain MRI is 3D, the slice-based 2D models analyze slices of one subject independently, leading to the loss of structural information (Khodatars et al. 2020; Wen et al. 2020). Moreover, the conversion leads to the loss of interslice information, resulting in a suboptimal accuracy of 61.83%. In volume-based 3D DenseNet, we use a 3D model with the same depth as the 2D model and get an accuracy of 62.39%. Mix-up is also adopted in the 3D model. Compared with these 2 methods, our method achieves significant performance improvement of 6.02% and 5.46%, powerfully demonstrating the superiority of our framework. Moreover, we train a 3D DenseNet using 3D cortical shape metrics to further examine the influence of different distance measurement methods. We resample the 3D surface meshes of each shape metric (thickness, sulcal depth, and curvature) into the 3D matrix, and then we train 3D DenseNets on the obtained 3D matrices. We use the same training strategies as the 2D models, and the mix-up and stacking are used for a fair comparison. The 3D model uses the Euclidean distance directly and achieves an accuracy of 61.13%, which is lower than our framework. 这些方法的结果如表2所示。支持向量机是脑磁共振成像的经典方法之一。这些数据被预处理并重新形成一个矢量，并采用主成分分析(PCA)(Wold et al. 1987)进行特征提取。支持向量机算法的准确率为58.12% ，明显低于所提出的两阶段迁移学习算法，说明了该算法的优越性。一些研究使用2D CNN 通过将大脑图像切成片来分析它们。类似地，在基于切片的 DenseNet 中，将3D 脑图像分成3个方向的切片，并使用获得的切片对预先训练好的模型进行微调。切片的结果最终与堆叠结合在一起。采用混合学习、叠加学习和迁移学习策略，以确保公平比较。尽管大脑 MRI 是3D 的，但基于切片的2D 模型独立分析一个受试者的切片，导致结构信息的丢失(Khodatars 等，2020; Wen 等，2020)。而且，这种转换会导致切片间信息的丢失，从而导致61.83% 的次优准确率。在基于体积的3D DenseNet 中，我们使用了与2D 模型具有相同深度的3D 模型，得到了62.39% 的精度。在三维模型中也采用了混合处理。与上述两种方法相比，本文提出的方法在性能上分别取得了6.02% 和5.46% 的显著提高，有力地证明了本文框架的优越性。此外，我们利用三维皮层形状指标训练一个三维致密网，以进一步检验不同距离测量方法的影响。我们将每个形状度量(厚度、沟深和曲率)的三维表面网格重采样到三维矩阵中，然后在得到的三维矩阵上训练三维 DenseNet。我们使用相同的训练策略作为二维模型，混合和堆叠是用于一个公平的比较。这个3 d 模型直接使用了欧几里得度量，达到了61.13% 的准确率，比我们的框架要低。 Ablation Study消融研究We investigate the effectiveness of each module in our model with an ablation study (Table 3). DenseNet is adopted as the base model. The stacking and mix-up strategies are investigated. We combine stacking and mix-up with the base model to get new models. Then the acquired models are tested with the ABIDE dataset. The stacking and mix-up bring improvements of 1.50% and 0.70% accuracy, respectively, demonstrating the effectiveness of the stacking and mix-up training strategies. 我们通过消融研究调查了模型中每个模块的有效性(表3)。基本模型采用了致密网络(DenseNet)。研究了叠加和混合策略。我们结合叠加和混合的基础模型得到新的模型。然后使用 ABIDE 数据集测试获取的模型。叠加训练和混合训练的准确率分别提高了1.50% 和0.70% ，证明了叠加训练和混合训练策略的有效性。 Table 3 表3 Results of ablation study
All experiments are based on the proposed 2-stage transfer learning framework. The average of all metrics is used as the result when the stacking is not adopted. The best results are shown in bold. Table 3 表3 Results of ablation study 消融研究结果
All experiments are based on the proposed 2-stage transfer learning framework. The average of all metrics is used as the result when the stacking is not adopted. The best results are shown in bold. 所有的实验都基于所提出的两阶段迁移学习框架。如果没有采用堆叠，则使用所有度量的平均值作为结果。最佳结果以粗体显示。 Visualization Results视像化结果As mentioned above, we utilize the occlusion test to visualize the critical regions for sex classification and ASD classification. We average the results of all subjects to obtain the group-level differences. The results of thickness, sulcal depth, curvature, and myelin map are calculated, respectively. The most critical regions are shown in Figure 4. The red color indicates highly discriminative brain regions, whereas the blue color denotes less discriminative regions. 如上所述，我们利用遮挡测试来可视化性别分类和 ASD 分类的关键区域。我们对所有受试者的结果进行平均，以获得群体水平的差异。分别计算了厚度、沟深、曲率和髓鞘图。最关键的区域如图4所示。红色表示高度区分的大脑区域，而蓝色表示较少区分的区域。 Figure 4 图4 Visualization of discriminative regions for sex classification and ASD classification using occlusion test. The most discriminative brain regions are marked. The red regions contribute more to classification. 基于遮挡试验的性别分类和 ASD 分类区域可视化研究。大脑中最具辨别力的区域被标记出来。红色区域对分类的贡献更大。 Discussion讨论Methodology研究方法In this study, we use deep learning for MRI imaging analysis for several reasons. As the size of the dataset increases, the representation ability of traditional machine learning methods has reached limits. On the contrary, CNN models have many parameters to be trained and show better representational capability of fitting high-dimensional data such as brain images. Traditional machine learning algorithms like SVM usually depend on hand-crafted features. CNN models can automatically extract local features and distributed representations without complicated feature engineering, which is of great significance for finding brain differences and searching neurological biomarkers. Moreover, deep learning can be adapted to different tasks due to its strong adaptability. The transfer learning techniques for deep learning are matured and thus the pretrained models can be used small sample tasks for better performance. 在本研究中，我们使用深度学习进行 MRI 成像分析有几个原因。随着数据集规模的增大，传统机器学习方法的表示能力已经达到了极限。相反，细胞神经网络模型有许多参数需要训练，并显示出更好的表征能力来拟合高维数据，如脑图像。像 SVM 这样的传统机器学习算法通常依赖于手工制作的特性。细胞神经网络模型可以自动提取局部特征和分布式表征，而不需要复杂的特征工程，这对于发现大脑差异和寻找神经生物标志物具有重要意义。此外，深度学习具有很强的适应性，可以适应不同的任务。用于深度学习的迁移学习技术已经成熟，因此预先训练的模型可以用于小样本任务以获得更好的性能。 Mapping the 3D cerebral cortex as 2D images also brings several benefits. The complex geometry and folding patterns of the cerebral cortex hinder its analysis. For example, 2 adjacent voxels in Euclidean space may be anatomically or functionally segregated due to the non-Euclidean geometry of the folding cerebral cortex. Current CNN models are based on Euclidean distance and ignore the structural features of the cerebral cortex, which is straightforward but coarse. The signatures from different brain regions are mixed after the convolutional operations, which is harmful in searching diagnostic biomarkers. Different from current deep learning models, the proposed framework uses the distance along the cortical surface to measure the relative positions of different brain regions, which is more neurobiologically relevant. Compared with other models based on Euclidean distance, our framework could better preserve the structural layout of the brain and obtain results with neurobiological significance. Moreover, many 2D pretrained models are available for transfer learning under our framework, which can significantly improve network performance. It is also evident that the converted 2D images are substantially compressed from 3D raw data while keeping the most valid information, leading to high efficiency during model training. Freesurfer also provides a 2D mapping, but the topological structure is not preserved because the mapped cerebral cortex is torn. The mapped 2D image of Freesurfer is irregular, which is harmful in the training of deep models. Moreover, even though we use the Freesurfer for preprocessing in this paper, our framework is also compatible with other preprocessing tools such as CIVET (MacDonald et al. 2000) and Fastsurfer (Henschel et al. 2020) only if they can generate similar cortical surface meshes and vertex-wise shape metrics. 将3D 大脑皮层映射为2D 图像也会带来一些好处。大脑皮层的复几何和折叠模式妨碍了它的分析。例如，由于折叠的大脑皮层的非欧几里得几何，欧几里得空间中相邻的两个体素可能在解剖学上或功能上被隔离。目前的 CNN 模型是基于欧几里得度量的，忽略了大脑皮层的结构特征，大脑皮层虽然简单但是粗糙。不同脑区的特征信号在卷积术后混杂在一起，不利于寻找诊断性的生物标志物。与目前的深度学习模型不同，提出的框架使用沿皮层表面的距离来测量不同大脑区域的相对位置，这在神经生物学上更相关。与其他基于欧几里得度量的模型相比，我们的框架能够更好地保存大脑的结构布局，并获得具有神经生物学意义的结果。此外，在我们的框架下，许多二维预训练模型可用于迁移学习，这可以显著提高网络性能。显然，转换后的二维图像基本上是从三维原始数据压缩而来的，同时保留了最有效的信息，从而提高了模型训练的效率。Freesurfer 还提供了一个2D 映射，但是由于映射的大脑皮层被撕裂，拓扑结构没有得到保留。自由冲浪运动员的二维映射图像不规则，不利于深度模型的训练。此外，即使我们在本文中使用 Freesurfer 进行预处理，我们的框架也与 CIVET (MacDonald et al。2000)和 Fastsurfer (Henschel et al。2020)等其他预处理工具兼容，只要它们能够生成类似的皮层表面网格和顶点形状指标。 Transfer Learning From ImageNet从 ImageNet 迁移学习The training of deep models requires a large-scale dataset, which is troublesome in brain imaging analysis since the collection of data is expensive and time-consuming. Many studies use models pretrained on natural image to medical image classification and achieve success. However, natural images are 2D planar ones, whereas brain images are always 3D, hindering the application of transfer learning in brain imaging studies. Our solution is to transform the cerebral cortex into 2D images and make transfer learning from natural images to brain images applicable. In this study, the models are first pretrained on a large natural image dataset, that is, ImageNet, and then fine-tuned with the converted 2D brain images. We demonstrate the effectiveness of transfer learning from natural images to MRI data by a robust and significant performance improvement of both sex classification and ASD classification. The success of the proposed deep transfer learning framework is expected and reasonable. Even though brain images are different from natural images, the pretrained model can provide universal features, especially low-level features, which can be effectively reused in brain imaging analysis. In the converted 2D brain images, morphometric features such as sulcal depth can be regarded as a kind of textural feature in the image processing field. The models pretrained on natural images have learned a mass of knowledge on texture features and are excellent in extracting texture features, which is helpful in the training of the brain morphometric features. Transfer learning also provides models with suitable initial parameters and makes the network easier to converge for small-scale datasets. 深度模型的训练需要大规模的数据集，由于数据的收集成本高、耗时长，给脑成像分析带来了很大的困难。许多研究利用基于自然图像的预训练模型对医学图像进行分类，并取得了成功。然而，自然图像是二维平面图像，而大脑图像往往是三维的，阻碍了转移学习在脑成像研究中的应用。我们的解决方案是将大脑皮层转换成二维图像，并使从自然图像到大脑图像的转移学习适用。在这项研究中，模型首先在一个大型自然图像数据集，即 ImageNet 上进行预训练，然后用转换后的2D 脑图像进行微调。我们证明了从自然图像到 MRI 数据的转移学习的有效性，通过性别分类和 ASD 分类的强大和显着的性能改进。所提出的深度迁移学习框架的成功预期是合理的。尽管脑图像不同于自然图像，但预训练模型可以提供通用特征，特别是低层特征，可以有效地重用于脑图像分析。在转换后的二维脑图像中，脑沟深度等形态测量特征可以看作是图像处理领域的一种纹理特征。在自然图像上预训练的模型学习了大量关于纹理特征的知识，在提取纹理特征方面表现出色，有助于训练大脑形态特征。传递学习还为模型提供了合适的初始参数，使得小规模数据集的网络收敛更加容易。 Two-Stage Transfer Learning两阶段迁移学习The ABIDE dataset used in this study is collected from over 30 sites. The differences in scanning machines and experimental parameters usually introduce intersite data heterogeneity (Chen et al. 2015). Compared with single-site ASD classification studies, such multisite studies are more difficult (Katuwal et al. 2016). In contrast with many other studies that use part of the ABIDE dataset, our study involves all available subjects from ABIDE-I and ABIDE-II, which is more challenging but fairer. We have demonstrated that transfer learning from natural image classification to ASD classification works. To further improve the performance on ABIDE, we propose a 2-stage transfer learning framework using the HCP as an intermediate domain and achieve higher accuracy than direct transfer learning from ImageNet. HCP is an excellent intermediate dataset due to its good data quality and better homogeneity. As aforementioned, direct transfer learning from ImageNet could provide useful low-level features. Compared with ImageNet, HCP is more similar to the data of ABIDE. A reasonable guess is that the data distribution inconsistency between natural images and brain images of ASD can be alleviated with brain images of healthy people. The fine-tuning on HCP makes the features more suitable for the ASD classification. To validate our hypothesis, we calculate the filter variation of different convolutional layers (Supplementary Fig. S3). Compared with direct transfer learning, the second-stage transfer learning from HCP to ABIDE shows a smaller variation in both low-level and high-level filters, suggesting that the first-stage transfer learning brings benefits in both low-level and high-level features. Moreover, sex classification is an excellent intermediate task. It is well known that sex is closely related to some neuropsychiatric disorders. For example, ASD appears 4 times greater in males than females. Previous studies have investigated the sex differences in ASD and emphasized the critical role of sex in ASD studies (Lawrence et al. 2020). The models fine-tuned on the HCP dataset have learned sex-related features, which may be helpful in the classification of ASD, considering the sex-biased phenomenon of the disorder. The doubtless sex label of HCP also ensures the reliability of the first-stage transfer learning. Moreover, we evaluate the effect of age by grouping the ABIDE into 2 groups based on whether the age of the sample is covered by HCP. There is no significant difference in performance improvement for 2-stage transfer learning between 2 groups. Our 2-stage transfer learning provides a new approach for solving the small sample problem in the classification of neuropsychiatric disorders by using both natural images and other brain imaging data. Important Regions for Sex and ASD Classification性别和 ASD 分类的重要区域The brain differences between males and females have been explored in many previous studies. In this study, we find some critical regions in the sex classification based on shape metrics, that is, thickness, sulcal depth, curvature, and myelin map. The occlusion test maps directly visualize the importance of different brain regions in the sex classification. As shown in Figure 4, we get 4 different occlusion test maps for the brain shape metrics, whereas some metric-shared discriminative regions exist. We find that the superior frontal cortex, superior parietal cortex, supramarginal cortex, paracentral cortex, precuneus, temporal pole, and right lingual cortex exhibit higher discriminative power in the sex classification, which have also been reported in the previous studies with different imaging modalities. One of our previous studies has reported the sex-related structural and functional differences in the superior frontal cortex, right supramarginal cortex, right lingual cortex, and left superior parietal cortex (Wang et al. 2012). The gray matter volume differences of the precuneus and temporal pole are demonstrated in previous studies (Ruigrok et al. 2014). The sex differences of the superior frontal cortex, superior parietal cortex, and paracentral cortex are also observed during the risk-taking tasks (Lee et al. 2009). The identified brain regions are closely related to behavioral or cognitive differences between males and females. The superior frontal cortex is a vital brain region involved with various cognitive and motor tasks, including working memory, self-awareness, and attention (Li et al. 2013). The superior parietal cortex mainly focuses on visuospatial and attention processing, long-term and working memory (Koenigs et al. 2009). The lingual cortex is closely related to vision tasks and word processing (Mechelli et al. 2000). Precuneus participates in visuospatial imagery, episodic memory retrieval, and self-processing operations (Cavanna and Trimble 2006). The temporal pole is linked to social and emotional processing (Snowden et al. 2004). The paracentral cortex controls the motor and sensory innervations of the lower extremity, including muscles and the urinary bladder (Spasojević et al. 2013). ASD is a developmental disorder characterized by difficulty in social interactions, verbal and nonverbal communication deficits, and stereotyped activities and limited interests (Lord et al. 2018). In this study, we find that the superior frontal cortex, precentral cortex, postcentral cortex, inferior temporal cortex, middle temporal cortex, left superior temporal cortex, and right fusiform are critical regions in the classification of ASD in more than 2 metrics. These regions are also investigated in other studies. For example, the curvature and folding index features from frontal and temporal cortices are dominant in the early detection of ASD (Katuwal et al. 2016). Differences in the right inferior temporal cortex and right fusiform are reported between ASD patients and normal controls (Shahamat and Abadeh 2020). A functional magnetic resonance imaging (fMRI) study reveals group differences in the development of the superior temporal cortex (Prigge et al. 2013). The change of postcentral cortex, precentral cortex, and superior frontal cortex is also reported (Chen et al. 2015). The frontal cortex is thought to be related to high-order cognition, social and emotional functions, language, which are deficient in ASD (Carper and Courchesne 2005). Some studies report motor function abnormalities in ASD, which is regarded to be related to the precentral and postcentral (Müller et al. 2001). Temporal regions are related to social perception, language, and the “theory of mind,” which are impaired in ASD (Gendry Meresse et al. 2005). Fusiform plays an essential role in face perception, which is the key feature of normal social functioning in humans. However, the fusiform cortex is found hypoactive in patients with ASD, which cause the abnormalities in face perception and social interactions (van Kooten et al. 2008). 自闭症是一种发展障碍拥有属性的社交障碍，缺乏语言和非言语交际，活动定型和兴趣有限(Lord et al。2018)。在本研究中，我们发现上额叶皮层，中央前皮层，中央后皮层，下颞叶皮层，中颞叶皮层，左上颞叶皮层和右梭形是 ASD 分类的关键区域，超过2个指标。其他研究也对这些区域进行了调查。例如，额叶和颞叶皮层的曲率和折叠指数特征在 ASD 的早期发现中占主导地位(Katuwal 等，2016)。报道了 ASD 患者与正常对照者的右颞下皮质和右梭形区域的差异(Shahamat 和 Abadeh 2020)。一项功能性磁共振成像成像(fMRI)研究揭示了颞上皮质发育的群体差异(Prigge et al. 2013)。中央后皮层，中央前皮层和额上皮层的变化也有报道(Chen et al。2015)。额叶皮层被认为与高级认知、社会和情感功能、语言有关，这些都是 ASD 所缺乏的(Carper and Courchesne 2005)。一些研究报道 ASD 的运动功能异常，被认为与中枢前后相关(Müller et al. 2001)。时间区域与社会知觉、语言和“心理理论”有关，这些在 ASD 中受到损害(Gendry Meresse et al. 2005)。梭形在人的面孔感知中起着重要作用，是人类正常社会功能的关键特征。然而，梭形皮质在 ASD 患者中被发现是低活性的，这导致了面部感知和社会交往的异常(van Kooten et al. 2008)。 Our results are consistent with the conclusions of previous sex and ASD studies, validating the reliability of our results. It should also be noted that the brain regions with group differences are observed in different metric maps, indicating that the alterations of these brain regions are stable. 我们的结果与之前的性别和自闭症研究的结论一致，验证了我们结果的可靠性。还应该注意的是，在不同的度量图中可以观察到具有群体差异的大脑区域，这表明这些大脑区域的改变是稳定的。 Limitations and Future Work限制和未来的工作Although the proposed framework has made some progress, there are still several limitations. Firstly, there is a lack of fair comparison with other studies due to different sample sizes and brain features. Secondly, the cerebellum and subcortical regions are not involved in the analysis, with which the classification performance may be further improved. Recent studies have shown that the cerebellum and subcortical regions can also be converted into surface meshes (Chye et al. 2019; Sereno et al. 2020), and we will follow-up on the relevant studies and further complete our framework. Thirdly, we only use structural MRI in this study, but the proposed framework can be further extended to functional MRI. The combined structural and functional MRI may achieve better performance. We will promote our framework into fMRI data in the future. 虽然提出的框架已经取得了一些进展，但仍然存在一些局限性。首先，由于样本量和大脑特征的不同，缺乏与其他研究的公平比较。其次，不涉及小脑和皮质下区域，这样可以进一步提高分类性能。最近的研究表明，小脑和皮质下区域也可以转化为表面网格(Chye 等，2019; Sereno 等，2020) ，我们将跟进相关研究并进一步完善我们的框架。第三，在本研究中我们只使用结构性磁共振成像，但是所提出的框架可以进一步扩展到功能性磁共振成像。结构和功能相结合的 MRI 可以获得更好的性能。我们将在未来将我们的框架推广到功能磁共振成像数据。 Conclusion结论In this paper, we propose a framework to map the 3D cerebral cortex into 2D images with geometry mapping and facilitate transfer learning from natural images to brain images. In this way, the mature algorithm and techniques for 2D images in computer vision can be easily applied in brain image analysis. The topological information of brain structure is preserved, which is plausible for cortical visualization and neurobiological analysis. We validate the effectiveness of our framework on sex and ASD classification with both traditional transfer learning and a novel 2-stage transfer learning and achieved significant performance improvement. The proposed framework creatively applies 2D pretrained models to cortical shape-based classification, shedding new light for brain image analysis. 在本文中，我们提出了一个框架来映射三维大脑皮层到二维图像的几何映射，并促进从自然图像到大脑图像的转移学习。这样，成熟的计算机视觉二维图像处理算法和技术就可以很容易地应用于脑图像分析。保留了大脑结构的拓扑信息，可用于皮层可视化和神经生物学分析。我们通过传统的迁移学习和一种新的两阶段迁移学习验证了我们的框架在性别和 ASD 分类上的有效性，并取得了显著的性能改善。该框架创造性地将二维预训练模型应用于基于皮层形状的分类，为脑图像分析提供了新的思路。 Notes笔记Conflict of Interest: None declared. 利益冲突: 未申报。 Funding资金National Key Research and Development Program (grant 2018YFB1305101); the National Natural Science Foundation of China (grant 62036013, 61722313, 61773391); the Science & Technology Innovation Program of Hunan Province (grant 2018RS3080). 国家重点研究开发项目(赠款2018YFB1305101) ; 国家自然科学基金(赠款62036013,61722313,61773391) ; 湖南省科技创新项目(赠款2018RS3080)。 |
|
|
PDF
