Contributions
Abstract: EP1444
Type: ePoster
Abstract Category: Pathology and pathogenesis of MS - Imaging
Background: Volumetric assessment of deep grey matter structures is of increasing relevance for the assessment of degenerative changes linked to MS progression. Automated segmentation methods are typically validated on manual outlines, however, defining standard operating procedures for anatomically accurate and reproducible outlining remains challenging and cumbersome to evaluate.
Objectives: To assess anatomical consistency and volumetric reproducibility of raters guided by a detailed segmentation protocol for outlining the caudate (C), putamen (P), and thalamus (T) using a new, web-based virtual laboratory.
Methods: Three independent raters traced C, P, and T on 3T T1-weighted images (multi-site) from MS patients and healthy controls following a consensus protocol with detailed instructions and illustrations to guide in the definition of anatomical boundaries. SPINE, a web-based “virtual laboratory” that enables tracing in a web-browser, as well as interactive exploratory analysis (numerical and visual inspection) was used. Intra- and inter-rater reliability was assessed using intraclass correlation [consistency-of-agreement (CA-ICC) and absolute-agreement (AA-ICC)]; Bland-and Altman analysis, spatial overlap (Dice coefficients), as well as interactive visual inspection of anatomical accuracy and consistency.
Results: Inter-rater CA-ICC (mean (95%CI)) was 0.98 (0.49-1.00) (T), 0.99 (0.91-1.00) (C) and 0.98 (0.81-1.00) (P). Inter-rater AA-ICC was 0.99 (0.81-1.00) (T), 0.76 (0.07-0.99) (C) and 0. 78 (0.09-0.99) (P).
Interestingly, the student"s, medical doctor"s, and senior Neurologist"s intra-rater bias according to Bland-Altman was 30.7%, -14.2%, and 5.7% (T), respectively; 17.1%, 14.5%, and -2.6 (P); 22.6%, 9.5%, and -1.2% (C), suggesting that intra-rater reproducibility was impacted by the degree of radiological experience. Raters were consistently more reproducible on C, followed by P, with T exhibiting the worst reproducibility.
Overlap analysis signalled good inter-rater reproducibility (Mean Dice coefficients: 0.82 (T), 0.81 (C), 0.81 (P).
Nevertheless, interactive visual inspection of pair-wise comparisons between raters revealed that outlining errors were not equally distributed around the structures of interest, and demonstrated significant anatomical inconsistencies in all three structures.
Conclusion: Reliance on ICC and overlap analysis is insufficient and potentially misleading in assessing accuracy and reproducibility of anatomical parcellation.
Disclosure: J. Burggraaff received funding for research from the ´Nauta Fonds´
J.C. Prieto has nothing to disclose
J.P. Simoes has nothing to disclose
Y. Liu was a recipient of the ECTRIMS-MAGNIMS fellowship
S. Ruggieri has nothing to disclose
M. Palotai is currently a recipient of the McDonald Fellowship from the Multiple Sclerosis International Federation
C. Gasperini received fee as speaker for Bayer-Schering Pharma, Sanofi-Aventis, Genzyme, Biogen, Teva, Novartis, Merck Serono. Received a grant for research by Teva.
M.P. Wattjes serves on the editorial boards of Neuroradiology, Journal of Neuroimaging, European Radiology, Frontiers of Neurology, and serves as a consultant for Roche, Novartis and Biogen.
F. Barkhof serves on the editorial boards of Brain, Neurology, Neuroradiology, Multiple Sclerosis Journal and Radiology, and serves as a consultant for Bayer-Schering Pharma, Sanofi-Aventis, Genzyme, Biogen, Teva, Novartis, Roche, Synthon BV and Jansen Research.
H. Vrenken has received research grants from Pfizer, MerckSerono, Novartis and Teva, and a speaker honorarium from Novartis. All funds were paid directly to his institution.
C.R.G. Guttmann has nothing to disclose that could constitute a potential conflict of interest for this work. Received a grant for research from Sanofi and from the National Multiple Sclerosis Society.
Abstract: EP1444
Type: ePoster
Abstract Category: Pathology and pathogenesis of MS - Imaging
Background: Volumetric assessment of deep grey matter structures is of increasing relevance for the assessment of degenerative changes linked to MS progression. Automated segmentation methods are typically validated on manual outlines, however, defining standard operating procedures for anatomically accurate and reproducible outlining remains challenging and cumbersome to evaluate.
Objectives: To assess anatomical consistency and volumetric reproducibility of raters guided by a detailed segmentation protocol for outlining the caudate (C), putamen (P), and thalamus (T) using a new, web-based virtual laboratory.
Methods: Three independent raters traced C, P, and T on 3T T1-weighted images (multi-site) from MS patients and healthy controls following a consensus protocol with detailed instructions and illustrations to guide in the definition of anatomical boundaries. SPINE, a web-based “virtual laboratory” that enables tracing in a web-browser, as well as interactive exploratory analysis (numerical and visual inspection) was used. Intra- and inter-rater reliability was assessed using intraclass correlation [consistency-of-agreement (CA-ICC) and absolute-agreement (AA-ICC)]; Bland-and Altman analysis, spatial overlap (Dice coefficients), as well as interactive visual inspection of anatomical accuracy and consistency.
Results: Inter-rater CA-ICC (mean (95%CI)) was 0.98 (0.49-1.00) (T), 0.99 (0.91-1.00) (C) and 0.98 (0.81-1.00) (P). Inter-rater AA-ICC was 0.99 (0.81-1.00) (T), 0.76 (0.07-0.99) (C) and 0. 78 (0.09-0.99) (P).
Interestingly, the student"s, medical doctor"s, and senior Neurologist"s intra-rater bias according to Bland-Altman was 30.7%, -14.2%, and 5.7% (T), respectively; 17.1%, 14.5%, and -2.6 (P); 22.6%, 9.5%, and -1.2% (C), suggesting that intra-rater reproducibility was impacted by the degree of radiological experience. Raters were consistently more reproducible on C, followed by P, with T exhibiting the worst reproducibility.
Overlap analysis signalled good inter-rater reproducibility (Mean Dice coefficients: 0.82 (T), 0.81 (C), 0.81 (P).
Nevertheless, interactive visual inspection of pair-wise comparisons between raters revealed that outlining errors were not equally distributed around the structures of interest, and demonstrated significant anatomical inconsistencies in all three structures.
Conclusion: Reliance on ICC and overlap analysis is insufficient and potentially misleading in assessing accuracy and reproducibility of anatomical parcellation.
Disclosure: J. Burggraaff received funding for research from the ´Nauta Fonds´
J.C. Prieto has nothing to disclose
J.P. Simoes has nothing to disclose
Y. Liu was a recipient of the ECTRIMS-MAGNIMS fellowship
S. Ruggieri has nothing to disclose
M. Palotai is currently a recipient of the McDonald Fellowship from the Multiple Sclerosis International Federation
C. Gasperini received fee as speaker for Bayer-Schering Pharma, Sanofi-Aventis, Genzyme, Biogen, Teva, Novartis, Merck Serono. Received a grant for research by Teva.
M.P. Wattjes serves on the editorial boards of Neuroradiology, Journal of Neuroimaging, European Radiology, Frontiers of Neurology, and serves as a consultant for Roche, Novartis and Biogen.
F. Barkhof serves on the editorial boards of Brain, Neurology, Neuroradiology, Multiple Sclerosis Journal and Radiology, and serves as a consultant for Bayer-Schering Pharma, Sanofi-Aventis, Genzyme, Biogen, Teva, Novartis, Roche, Synthon BV and Jansen Research.
H. Vrenken has received research grants from Pfizer, MerckSerono, Novartis and Teva, and a speaker honorarium from Novartis. All funds were paid directly to his institution.
C.R.G. Guttmann has nothing to disclose that could constitute a potential conflict of interest for this work. Received a grant for research from Sanofi and from the National Multiple Sclerosis Society.