Contributions
Abstract: P1796
Type: Poster Sessions
Abstract Category: N/A
Introduction: Current practice for predicting disability progression relies on linear regression (LR), which produces only modest correlations, possibly due to its linear limitations. Machine learning can model relationships between variables that may or may not follow expected linear trends. We propose a machine learning method using a random forest (RF) for predicting disability progression in secondary progressive multiple sclerosis (SPMS).
Objective: To evaluate the performance of RF for predicting confirmed disability progression (CDP) and no CDP within 1-year in SPMS patients compared to LR, using baseline clinical and magnetic resonance imaging (MRI) data.
Methods: Data from a placebo-controlled (negative) SPMS trial assessing MBP8298 included expanded disability status scale (EDSS), multiple sclerosis functional composite, including timed 25-foot walk, 9-hole peg test, paced auditory serial addition, disease duration, age, collected every 3 months, and MRI features (brain volume and T2 lesion volume). CDP was defined as an increase in EDSS (≥1.0 and ≥0.5 for baseline ≤5.5 and ≥6.0 respectively) sustained for 6 months, originating within 12 months of baseline.
Training and validation of RF and LR were done using 10 stratified 10-fold cross validations with week 0 data as baseline to predict CDP by week 52.
Validation testing of RF and LR were compared for sensitivity, specificity, accuracy in predicting CDP (positive predictive value) and no CDP (negative predictive value), and ability to separate CDP from no CDP (area under the receiver-operator characteristic curve, AUC). Results were averaged across repetitions.
Results: 78 of 418 (19%) SPMS patients progressed within 1 year of week 0, and 340 (81.3%) did not. RF and LR performance were sensitivity (%): 56.5 vs. 36.6; specificity (%): 75.4 vs. 59.4; positive predictive value (%): 35.5 vs. 17.1; negative predictive value (%): 88.4 vs. 80.3; AUC (%) 66.0% vs. 48.0. P< 0.0001 for all comparisons. LR performed worse than random guessing (AUC=50%).
Conclusion: Machine learning using RF outperforms the traditional LR approach for predicting CDP and no CDP in SPMS. While positive predictive value is low, the ability to predict SPMS patients who are unlikely to progress during the next 12 months may improve the efficiency of recruitment for phase II clinical trial design for progressive multiple sclerosis.
Disclosure: Marco T. K. Law has nothing to disclose. Anthony L. Traboulsee has received grant funding from the MS Society of Canada, Canadian Institute for Health Research, Roche, and Genzyme; received honoraria or travel grants from Teva Canada Innovation, Roche, Genzyme, Chugai Pharmaceuticals. David K. B. Li has received research funding from the Canadian Institute of Health Research and Multiple Sclerosis Society of Canada. He is the Emeritus Director of the UBC MS/MRI Research Group which has been contracted to perform central analysis of MRI scans for therapeutic trials with Novartis, Perceptives, Roche and Sanofi-Aventis. The UBC MS/MRI Research Group has also received grant support for investigator-initiated independent studies from Genzyme, Merck-Serono, Novartis and Roche. He has acted as a consultant to Vertex Pharmaceuticals and served on the Data and Safety Advisory Board for Opexa Therapeutics and Scientific Advisory Boards for Adelphi Group, Celgene, Novartis and Roche. He has also given lectures which have been supported by non-restricted education grants from Biogen-Idec, Novartis, Sanofi-Genzyme and Teva. Robert Carruthers has received grants/research from MedImmune, Teva and Guthy Jackson; received speaking fees for unbranded lectures from Biogen, Genzyme and Teva and received consulting fees from Novartis, EMD Serono and Genzyme. Mark S. Freedmanhas received a research / educational grant from Genzyme; received honoraria or consultation fees from Actelion, BayerHealthcare, BiogenIdec, Chugai, Clene Nanomedicine, EMD Canada, Genzyme, Merck Serono, Novartis, Hoffman La-Roche, Sanofi-Aventis, Teva Canada Innovation; is member of a company advisory board, board of directors or other similar group of Actelion, BayerHealthcare, BiogenIdec, Hoffman La-Roche, Merck Serono, MedDay, Novartis, Sanofi-Aventis and is on speaker's bureau for Genzyme. Shannon Kolind has received a research / educational grant funding from Genzyme; received honoraria or consultation fees from Acorda and Genzyme; she is member of a company advisory board, board of directors or other similar group of Acorda and Genzyme. Roger Tam has received research support as part of sponsored clinical studies from Novartis, Roche, and Sanofi Genzyme.
Abstract: P1796
Type: Poster Sessions
Abstract Category: N/A
Introduction: Current practice for predicting disability progression relies on linear regression (LR), which produces only modest correlations, possibly due to its linear limitations. Machine learning can model relationships between variables that may or may not follow expected linear trends. We propose a machine learning method using a random forest (RF) for predicting disability progression in secondary progressive multiple sclerosis (SPMS).
Objective: To evaluate the performance of RF for predicting confirmed disability progression (CDP) and no CDP within 1-year in SPMS patients compared to LR, using baseline clinical and magnetic resonance imaging (MRI) data.
Methods: Data from a placebo-controlled (negative) SPMS trial assessing MBP8298 included expanded disability status scale (EDSS), multiple sclerosis functional composite, including timed 25-foot walk, 9-hole peg test, paced auditory serial addition, disease duration, age, collected every 3 months, and MRI features (brain volume and T2 lesion volume). CDP was defined as an increase in EDSS (≥1.0 and ≥0.5 for baseline ≤5.5 and ≥6.0 respectively) sustained for 6 months, originating within 12 months of baseline.
Training and validation of RF and LR were done using 10 stratified 10-fold cross validations with week 0 data as baseline to predict CDP by week 52.
Validation testing of RF and LR were compared for sensitivity, specificity, accuracy in predicting CDP (positive predictive value) and no CDP (negative predictive value), and ability to separate CDP from no CDP (area under the receiver-operator characteristic curve, AUC). Results were averaged across repetitions.
Results: 78 of 418 (19%) SPMS patients progressed within 1 year of week 0, and 340 (81.3%) did not. RF and LR performance were sensitivity (%): 56.5 vs. 36.6; specificity (%): 75.4 vs. 59.4; positive predictive value (%): 35.5 vs. 17.1; negative predictive value (%): 88.4 vs. 80.3; AUC (%) 66.0% vs. 48.0. P< 0.0001 for all comparisons. LR performed worse than random guessing (AUC=50%).
Conclusion: Machine learning using RF outperforms the traditional LR approach for predicting CDP and no CDP in SPMS. While positive predictive value is low, the ability to predict SPMS patients who are unlikely to progress during the next 12 months may improve the efficiency of recruitment for phase II clinical trial design for progressive multiple sclerosis.
Disclosure: Marco T. K. Law has nothing to disclose. Anthony L. Traboulsee has received grant funding from the MS Society of Canada, Canadian Institute for Health Research, Roche, and Genzyme; received honoraria or travel grants from Teva Canada Innovation, Roche, Genzyme, Chugai Pharmaceuticals. David K. B. Li has received research funding from the Canadian Institute of Health Research and Multiple Sclerosis Society of Canada. He is the Emeritus Director of the UBC MS/MRI Research Group which has been contracted to perform central analysis of MRI scans for therapeutic trials with Novartis, Perceptives, Roche and Sanofi-Aventis. The UBC MS/MRI Research Group has also received grant support for investigator-initiated independent studies from Genzyme, Merck-Serono, Novartis and Roche. He has acted as a consultant to Vertex Pharmaceuticals and served on the Data and Safety Advisory Board for Opexa Therapeutics and Scientific Advisory Boards for Adelphi Group, Celgene, Novartis and Roche. He has also given lectures which have been supported by non-restricted education grants from Biogen-Idec, Novartis, Sanofi-Genzyme and Teva. Robert Carruthers has received grants/research from MedImmune, Teva and Guthy Jackson; received speaking fees for unbranded lectures from Biogen, Genzyme and Teva and received consulting fees from Novartis, EMD Serono and Genzyme. Mark S. Freedmanhas received a research / educational grant from Genzyme; received honoraria or consultation fees from Actelion, BayerHealthcare, BiogenIdec, Chugai, Clene Nanomedicine, EMD Canada, Genzyme, Merck Serono, Novartis, Hoffman La-Roche, Sanofi-Aventis, Teva Canada Innovation; is member of a company advisory board, board of directors or other similar group of Actelion, BayerHealthcare, BiogenIdec, Hoffman La-Roche, Merck Serono, MedDay, Novartis, Sanofi-Aventis and is on speaker's bureau for Genzyme. Shannon Kolind has received a research / educational grant funding from Genzyme; received honoraria or consultation fees from Acorda and Genzyme; she is member of a company advisory board, board of directors or other similar group of Acorda and Genzyme. Roger Tam has received research support as part of sponsored clinical studies from Novartis, Roche, and Sanofi Genzyme.