SUPPLEMENTARY MATERIAL
Clinical prediction from structural brain MRI scans:
A large-scale empirical study
Mert R. Sabuncu1,2 and Ender Konukoglu1,
for the Alzheimer’s Disease Neuroimaging Initiative*
1 Athinoula A. Martinos Center for Biomedical Imaging, Harvard Medical School/Massachusetts General Hospital, Charlestown, MA
2 Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA
Supplementary Table S1
List of univariate measurements, such as thickness or volume of an anatomical ROI, that were most frequently associated with each variable of interest over the hundred 5-fold cross-validation sessions.
Supplementary Figure S1
Prediction performance metrics for each variable and MVPA algorithm, estimated via 5-fold cross-validation. (Panels a-c) Binary variables; (Panel d) Continuous variables. (Panel a) Area under the receiver operating characteristic curve (AUC) computed using MATLAB’s perfcurve tool (MATLAB 8.2, The Mathworks Inc. Natick, MA, 2013). (Panel b) True positive rate, or sensitivity. (Panel c) True negative rate, or specificity. (Panel d) Pearson’s correlation values. The MVPA algorithms are abbreviated as follows: N for neighborhood approximation forest, S for SVMs, and R for RVMs. The number after each letter denotes the feature type (1:aseg, 2:aparc, 3:aseg+aparc, 4:thick). The shaded gray color indicates statistical significance (-log10 p-value), where the p-value is computed via DeLong’s method and the colorbar is shown in Panel d. Associations with a p-value less than 0.01 are shown in red.
(a)
(b)
(c)
(d)
Supplementary Figure S2
The difference between the performance estimates of the three algorithms over all predicted variables and feature types. (Panel a) Correct Classification Ratio (CCR) differences with respect to the RVM, which has the highest average accuracy. All three classifiers are statistically equivalent. (Panel b) Normalized RMSE differences with respect to SVM, which offers the lowest average RMSE. NAF and SVM are statistically equivalent, and offer significantly better regression accuracy than RVM. On each box, the central mark is the median, the edges of the box are the 25th and 75th percentiles, the whiskers extend to the most extreme data points not considered outliers (which corresponds to a 99.3% data coverage if the data are normally distributed), and outliers are plotted individually. Plots were generated with Matlab’s boxplot function, with the ‘notch’ option turned on and default settings otherwise.
(a)
(b)
Supplementary Figure S3
The difference between the performance estimates of the MVPA models that employ the four image feature types, over all predicted variables and algorithms. (Panel a) Correct Classification Ratio (CCR) differences with respect to image feature 3 (aseg+aparc), which yields the highest average accuracy. Features 3 and 4 are statistically equivalent and offer slightly better accuracies than 1 and 2. (Panel b) Normalized RMSE differences with respect to Feature 4, which yields the lowest average NRMSE. Features 1 and 3 yield performances that are statistically equivalent to Feature 4. On each box, the central mark is the median, the edges of the box are the 25th and 75th percentiles, the whiskers extend to the most extreme data points not considered outliers (which corresponds to a 99.3% data coverage if the data are normally distributed), and outliers are plotted individually. Plots were generated with Matlab’s boxplot function, with the ‘notch’ option turned on and default settings otherwise.
(a)
(b)
Supplementary Figure S4
Algorithm versus feature range values. For each variable, we computed the algorithm range as the difference between the best and worst performance metrics (CCR for classification –Panel a- and NRMSE for regression –Panel b-) across the three algorithms (SVM, RVM and NAF), while fixing the feature type. These values were then averaged over feature types. Similarly, for each variable, the feature range was defined as the difference between the best and worst performance metrics across the four feature types, while fixing the algorithm type. These values were then average over the algorithms.
(a)
(b)
1