Contributions
Abstract: P880
Type: Poster
Abstract Category: Clinical aspects of MS - Clinical assessment tools
Background: In Multiple Sclerosis (MS) the Expanded Disability Status Scale (EDSS) is commonly used to assess disability but suffers from low inter- and intra-rater reliability. Supporting such assessments with computer systems based on machine learning is one possibility for improvement but require alternative methods to capture consistent clinical judgement. We recorded videos of patients performing the Finger-to-Nose Test (FNT) based on EDSS subscores for limb ataxia to use them as a reference.
Objective: To investigate whether the use of reference videos with predefined degrees of severity of limb ataxia can improve rating of other video clips.
Methods: Twenty-six neurologists from two participating centers rated 60 videos of MS patients performing FNT. Ratings were performed at baseline and after six weeks using the Neurostatus definitions of limb ataxia. The neurologists were randomized into two groups: one group used the reference videos for rating, the other rated without reference videos. The inter-rater consistency and long-term re-test agreement were determined.
Results: The use of reference videos made neurologists more consistent. We find that the spread of average ratings was lower when reference videos were used (standard deviation=0.12; range=0.40) than without reference videos (standard deviation=0.26; range=0.88), with a statistically significant difference in standard deviation (F-test; p=0.013). Similarly, use of reference videos increased the intraclass correlation coefficient for inter-rater agreement from 0.756 (without reference videos) to 0.816 (with reference videos). Long-term intra-rater agreement measured as percentage of identical ratings was similar between the two groups, with 69±11% without reference videos and 68±9% with reference videos.
Conclusions: The use of reference videos significantly improved consistency in rating upper limb ataxia by FNT. Hence, this may represent a method to rate movements in a more consistent way, particularly in the context of clinical research or for training of machine learning algorithms.
Disclosure:
M. DSouza: received travel support from Bayer AG, Teva and Genzyme and research support from the University Hospital Basel;
S. Steinheimer: received travel support from Bayer AG;
J. Dorn: employee of Novartis Pharma AG;
C. Morrison: is an employee of Microsoft Research.
J. Boisvert: employee of Novartis Pharma AG;
K. Miciunaite: none;
J. Burggraaff: received travel support from Novartis Pharma AG;
CEP van Munster: none;
P. Kontschieder: is an employee of Microsoft Research;
A. Sellen: is an employee of Microsoft Research;
CP Kamm: has received honoraria for lectures and consulting from Biogen-idec, Novartis, Bayer Schweiz AG, Teva, Merck-Serono, Genzyme and Pfize;
F. Dahlke: is an employee of Novartis Pharma AG.
B. Uitdehaag: received consultation fees from: Biogen Idec, Novartis, EMD Serono, Teva Pharmaceuticals, Genzyme and Roche. The Multiple Sclerosis Centre Amsterdam has received financial support for research, from Biogen Idec, Merck Serono, Novartis, and Teva Pharmaceuticals.
L. Kappos: his institution, the University Hospital Basel, has received research support and payments that were used exclusively for research support for my activities as principal investigator and member or, chair of planning and steering committees or advisory boards in trials sponsored by Actelion, Addex, Bayer Health Care Pharmaceuticals, Bayer Schering Pharma, Biogen Idec, CLC Behring, Genentech,, GeNeuro SA, Genzyme, Merck Serono, Mitsubishi Pharma, Novartis, Octapharma, Praxicon, Roche, Sanofi-Aventis, Santhera, Siemens and Teva. He has received grants from the Swiss MS Society, Swiss, National Research Foundation, the European Union, Gianni Rubatto Foundation, and the Novartis and Roche Research foundations.
Abstract: P880
Type: Poster
Abstract Category: Clinical aspects of MS - Clinical assessment tools
Background: In Multiple Sclerosis (MS) the Expanded Disability Status Scale (EDSS) is commonly used to assess disability but suffers from low inter- and intra-rater reliability. Supporting such assessments with computer systems based on machine learning is one possibility for improvement but require alternative methods to capture consistent clinical judgement. We recorded videos of patients performing the Finger-to-Nose Test (FNT) based on EDSS subscores for limb ataxia to use them as a reference.
Objective: To investigate whether the use of reference videos with predefined degrees of severity of limb ataxia can improve rating of other video clips.
Methods: Twenty-six neurologists from two participating centers rated 60 videos of MS patients performing FNT. Ratings were performed at baseline and after six weeks using the Neurostatus definitions of limb ataxia. The neurologists were randomized into two groups: one group used the reference videos for rating, the other rated without reference videos. The inter-rater consistency and long-term re-test agreement were determined.
Results: The use of reference videos made neurologists more consistent. We find that the spread of average ratings was lower when reference videos were used (standard deviation=0.12; range=0.40) than without reference videos (standard deviation=0.26; range=0.88), with a statistically significant difference in standard deviation (F-test; p=0.013). Similarly, use of reference videos increased the intraclass correlation coefficient for inter-rater agreement from 0.756 (without reference videos) to 0.816 (with reference videos). Long-term intra-rater agreement measured as percentage of identical ratings was similar between the two groups, with 69±11% without reference videos and 68±9% with reference videos.
Conclusions: The use of reference videos significantly improved consistency in rating upper limb ataxia by FNT. Hence, this may represent a method to rate movements in a more consistent way, particularly in the context of clinical research or for training of machine learning algorithms.
Disclosure:
M. DSouza: received travel support from Bayer AG, Teva and Genzyme and research support from the University Hospital Basel;
S. Steinheimer: received travel support from Bayer AG;
J. Dorn: employee of Novartis Pharma AG;
C. Morrison: is an employee of Microsoft Research.
J. Boisvert: employee of Novartis Pharma AG;
K. Miciunaite: none;
J. Burggraaff: received travel support from Novartis Pharma AG;
CEP van Munster: none;
P. Kontschieder: is an employee of Microsoft Research;
A. Sellen: is an employee of Microsoft Research;
CP Kamm: has received honoraria for lectures and consulting from Biogen-idec, Novartis, Bayer Schweiz AG, Teva, Merck-Serono, Genzyme and Pfize;
F. Dahlke: is an employee of Novartis Pharma AG.
B. Uitdehaag: received consultation fees from: Biogen Idec, Novartis, EMD Serono, Teva Pharmaceuticals, Genzyme and Roche. The Multiple Sclerosis Centre Amsterdam has received financial support for research, from Biogen Idec, Merck Serono, Novartis, and Teva Pharmaceuticals.
L. Kappos: his institution, the University Hospital Basel, has received research support and payments that were used exclusively for research support for my activities as principal investigator and member or, chair of planning and steering committees or advisory boards in trials sponsored by Actelion, Addex, Bayer Health Care Pharmaceuticals, Bayer Schering Pharma, Biogen Idec, CLC Behring, Genentech,, GeNeuro SA, Genzyme, Merck Serono, Mitsubishi Pharma, Novartis, Octapharma, Praxicon, Roche, Sanofi-Aventis, Santhera, Siemens and Teva. He has received grants from the Swiss MS Society, Swiss, National Research Foundation, the European Union, Gianni Rubatto Foundation, and the Novartis and Roche Research foundations.