Replication Procedure
Our goal, is to replicate the process of classifying MRI scans. We want to look past just Glial tumors and see if we can expand the field of Machine Learning based Radiomics with other MRI data sets. In particular, we used a data set from Kaggle on lung cancer MRI patient age prediction. [4] We did not decide to use the exact BraTS 2017 data set, due to it being unavailable as it was a part of a competition from two years ago.Thus, us choosing a free and open source data set. We have decided to continue using the method presented in the paper to see if their method could be applied to other data sets.
Original Experiment
Step 1:
Extract features from the dataset to describe long term or short term survival.
Step 2:
Create a prediction model from features extracted in step 1 to train the ML methods.
Step 3:
Evaluate the performance of each classifier by calculating the accuracy of each class separately and overall.
Our Experiment
Step 1:
Extract features from the Kaggle dataset to describe long term or short term survival. Our feature selection will involve deep features only to match the final results of the paper.
Step 2:
Create a prediction model from features extracted from Step 1 to train the ML methods.
Step 3:
Evaluate the performance of the dataset.
Implementation
To begin with building our model, we split the data set into groups based upon age ranges of 5 year increments, and contrasts of the image.
Next we used Tensorflow backend and Keras front end to build our AlexNet CNN. Some of our parameters and fine tuning of the CNN were chosen by what we thought made the most sense, since the source code of the paper we are basing this off was unavailable, and our data was slightly different.
The original data was given in the dicom file format, which we had to convert to PNGs and resize the images as per AlexNet's prerequisites. We then split the data into 11 labels, based on the age ranges of patients. We trained our model with a 80/20 split and ran the training and validation 20 times. As a reference, the loss function used is keras.losses.cateogrical_crossentropy and the model was optimized using the adam optimizer. The data used is the linked as [4] in the references (this is where Kaggle obtained the data from).
Results
After running our data set, we had an overall 30% accuracy on classifying ages of lung cancer patients in the best scenario, when running against the validation sets.
Below are our results of our runs.
Epoch 1/20
10/10 [==============================] - 8s 754ms/step - loss: 3.1477 - acc: 0.3200 - val_loss: 2.1999 - val_acc: 0.2000
Epoch 2/20
10/10 [==============================] - 6s 591ms/step - loss: 2.1903 - acc: 0.1800 - val_loss: 2.1500 - val_acc: 0.2000
Epoch 3/20
10/10 [==============================] - 6s 592ms/step - loss: 1.8676 - acc: 0.2400 - val_loss: 2.2112 - val_acc: 0.3000
Epoch 4/20
10/10 [==============================] - 6s 593ms/step - loss: 1.8031 - acc: 0.4200 - val_loss: 2.1678 - val_acc: 0.3000
Epoch 5/20
10/10 [==============================] - 6s 591ms/step - loss: 1.8048 - acc: 0.4000 - val_loss: 2.2280 - val_acc: 0.3000
Epoch 6/20
10/10 [==============================] - 6s 594ms/step - loss: 1.7760 - acc: 0.4000 - val_loss: 2.2664 - val_acc: 0.3000
Epoch 7/20
10/10 [==============================] - 6s 590ms/step - loss: 1.7250 - acc: 0.4200 - val_loss: 2.3182 - val_acc: 0.3000
Epoch 8/20
10/10 [==============================] - 6s 585ms/step - loss: 1.8025 - acc: 0.3800 - val_loss: 2.2130 - val_acc: 0.3000
Epoch 9/20
10/10 [==============================] - 6s 591ms/step - loss: 1.8520 - acc: 0.3800 - val_loss: 2.2741 - val_acc: 0.3000
Epoch 10/20
10/10 [==============================] - 6s 591ms/step - loss: 1.7504 - acc: 0.4200 - val_loss: 2.3055 - val_acc: 0.3000
Epoch 11/20
10/10 [==============================] - 7s 670ms/step - loss: 1.6482 - acc: 0.4000 - val_loss: 2.4272 - val_acc: 0.3000
Epoch 12/20
10/10 [==============================] - 6s 648ms/step - loss: 1.8151 - acc: 0.4000 - val_loss: 2.2875 - val_acc: 0.3000
Epoch 13/20
10/10 [==============================] - 7s 664ms/step - loss: 1.7880 - acc: 0.4000 - val_loss: 2.2820 - val_acc: 0.3000
Epoch 14/20
10/10 [==============================] - 6s 619ms/step - loss: 1.6871 - acc: 0.4400 - val_loss: 2.3939 - val_acc: 0.3000
Epoch 15/20
10/10 [==============================] - 7s 707ms/step - loss: 1.9223 - acc: 0.3400 - val_loss: 2.2529 - val_acc: 0.3000
Epoch 16/20
10/10 [==============================] - 7s 654ms/step - loss: 1.6706 - acc: 0.4200 - val_loss: 2.3574 - val_acc: 0.3000
Epoch 17/20
10/10 [==============================] - 6s 590ms/step - loss: 1.7550 - acc: 0.4000 - val_loss: 2.3793 - val_acc: 0.3000
Epoch 18/20
10/10 [==============================] - 6s 587ms/step - loss: 1.7329 - acc: 0.4200 - val_loss: 2.4151 - val_acc: 0.3000
Epoch 19/20
10/10 [==============================] - 6s 585ms/step - loss: 1.8789 - acc: 0.3200 - val_loss: 2.2852 - val_acc: 0.3000
Epoch 20/20
10/10 [==============================] - 6s 588ms/step - loss: 1.6658 - acc: 0.4600 - val_loss: 2.3883 - val_acc: 0.3000
{'val_loss': [2.1999414920806886, 2.1499626994132996, 2.2112219452857973, 2.167771279811859, 2.22795307636261, 2.2664164900779724, 2.318229389190674, 2.212981128692627, 2.274071145057678, 2.3054502248764037, 2.427194285392761, 2.2875027656555176, 2.2819632649421693, 2.3938969135284425, 2.252925395965576, 2.357350397109985, 2.379324221611023, 2.4151042342185973, 2.285162401199341, 2.388332450389862],
'val_acc': [0.20000000298023224, 0.20000000298023224, 0.30000000447034836, 0.3000000059604645, 0.30000000447034836, 0.3000000059604645, 0.30000000447034836, 0.3000000059604645, 0.3000000059604645, 0.30000000447034836, 0.3000000059604645, 0.3000000074505806, 0.3000000074505806, 0.30000000447034836, 0.3000000059604645, 0.3000000059604645, 0.30000000447034836, 0.3000000059604645, 0.3000000074505806, 0.3000000059604645],
'loss': [3.147661876678467, 2.190302872657776, 1.8675525903701782, 1.8030557870864867, 1.8048269391059875, 1.7759608030319214, 1.7250286102294923, 1.80247962474823, 1.8520369291305543, 1.7504295229911804, 1.6482178926467896, 1.8150883078575135, 1.7880383133888245, 1.6870878696441651, 1.9223135948181151, 1.670597541332245, 1.7549618124961852, 1.732863688468933, 1.8789369583129882, 1.6658036351203918],
'acc': [0.3200000047683716, 0.18000000417232515, 0.2400000050663948, 0.42000000923871994, 0.4000000074505806, 0.4000000089406967, 0.42000000923871994, 0.3800000101327896, 0.38000000864267347, 0.42000000923871994, 0.4000000074505806, 0.40000000447034834, 0.40000001192092893, 0.4400000095367432, 0.34000000506639483, 0.42000001221895217, 0.40000001043081285, 0.4200000137090683, 0.3200000047683716, 0.46000000834465027]}