Esra Zihni: Multimodal Classification using Neural Networks for Outcome Prediction in Acute-Ischemic Stroke

BCCN Berlin and TU Berlin

 

 

Abstract

Stroke is a major cause of death and long-term disabilities worldwide. Currently a time-window paradigm is being used as the decision criteria for thrombolysis treatment. Prediction of stroke outcome using machine learning could provide a decision support system for physicians and assist them in patient-oriented diagnosis and treatment. Patient-specific clinical parameters play an important role in outcome prediction. Integrating neuroimaging data that is acquired in the routine examination has the potential to improve the predictive accuracy. Our hypothesis is that using state-of-the-art deep learning algorithms to integrate both data modalities may significantly increase outcome prediction accuracy. In this project we developed automated methods to predict a binary 3 months post-stroke outcome using 3D time-of-flight (TOF) magnetic resonance angiography (MRA) images and clinical metadata. For this we designed four frameworks: 1) Multilayer perceptron (MLP) framework trained on clinical metadata and serve as the baseline model, 2) Convolutional neural net (CNN) framework trained on 3D TOF-MRA scans, 3) Feature extraction framework trained on the learned features extracted from the CNN and MLP models and 4) End-to-end framework trained on clinical metadata and 3D TOF images simultaneously using the fused CNN and MLP architectures respectively. For each framework five models were trained and evaluated on five different training-validation-test splits in order to account for the high data variability. The CNN and MLP models achieved average test AUC scores of 0.68 and 0.75 respectively. The feature extraction models achieved an average test AUC score of 0.75, whereas the end-to-end models achieved an average test AUC score of 0.76. Both multimodal frameworks showed significant improvement in the majority of the splits compared to the baseline MLP models. In this study we showed that integration of TOF-MRA imaging information with clinical data can potentially improve outcome prediction in stroke patients. Our established framework can be used to further integrate other imaging modalities with clinical metadata to improve predictive accuracy.

Master thesis defense

Organized by

Prof. Manfred Opper, Dr. Michelle Livne