Evaluation during different phases

For evaluation during phase 2:

• At the begining of phase 2, a preliminary test set will be released. Participants are required to predict the classes of this test set using the model trained in phase 1 (training phase).
• Weighted F1 scores will be computed using our evaluation script. This will decide the position of the respective participants/teams on the challenge leaderboard.
• Participants are also required to email a brief write-up explaining
• Methods used for data preprocessing, augmentation (if any)
• Description of the methodology used to solve the challenge for e.g. If using a deep learning method, a brief layout of the architecture should be included in the description.
• Challenges faced while data handling or any other kind of roadblocks.

• Based upon ranking in the leaderboard and submitted write-up, the ground truth of the preliminary test set will be released for the top 10% teams/individuals (20 max). This ground truth can be used by the participants for re-training/fine-tuning the model so as to give better performance on final test set. Finally, at the end of phase 2, pariticipants are required to submit a detailed paper in ISBI format with the following results on the preliminary test set and supplementary files conataing training curves and all the relevent results:

• Weighted-precision, weighted-recall and weighted-f1 score on both Normal and ALL class.
• Accuracy graphs on both Normal and ALL class.
• Subject level accuracies. For e.g. For patient X: 10 out of 15 predictions match the ground truth. So basically, Patient/subject level accuracy implies the percentage of correct predictions within a particular subject/patient.

For evaluation during phase 3:

• The final test set will be released for the participants who have submitted the detailed paper at the end of phase 2. Participants are required to upload a  file containing the predicted output labels for each test sample.
• Weighted F1 scores will be computed from the file using our evaluation script. This will decide the position of the respective participants/teams on the final challenge leaderboard.

The top three scorers on the leaderboard will be invited to attend the ISBI-2019 workshop. These participants will also be required to provide developed code and trained models to the organizing team.

Evaluation Metrics

Scikit-learn library is to be used for calculating weighted-precision, weighted-recall and weighted-f1 score. Please note that 'weighted' metrics will be used for performance evaluation purpose.

References: 
[1] Hammack, Daniel, and Julian de Wit. “Predicting Lung Cancer” blog.kaggle.com. http://blog.kaggle.com/2017/06/29/2017-data-science-bowl-predicting-lung-cancer-2nd-place-solution-write-up-daniel-hammack-and-julian-de-wit/ (Accessed 17th August 2018)

Rules

Please cite the following papers if you are using this challenge (or dataset) for any research purpose:

Publication Citation

This is a two class classification problem. The training and testing data contains folder level division of individual subjects. Example directory tree for data division is shown below:

Train folder:

• Patient 1
• Image1, Image2...
• Patient 1
• Image1, Image2...

Test folder:

• Patient 3
• Image1, Image2...
• Patient 4
• Image1, Image2...

The dataset contains a total of 118 individual subjects, distributed as follows:

• ALL (cancer) Subjects: 60 (Train + Preliminary test)
• Normal Subjects: 41 (Train + Preliminary test)

• Train set + Preliminary test composition: Total cells: 12528,   Cancer: 8491, Normal:  4037
• Final test set composition: Total cells: 2586

Data Availability Schedule: The data will be released in three phases.

Phase 1: During the first phase, we will release the training set. 

Phase 2: During the second phase, we will release the preliminary test set. 

Phase 3: During the third and final phase, we will release the final test set. 

The test set would only be made available to each participating team which submitted a detailed paper at the end of phase 2 (refer 'Evaluation' section). The participants would be given a limited time window (refer 'Important Dates' section) after the release of the test dataset. Participants are required to make the final submission within the given time frame.

What general pre-processing steps will be performed?

The data is already preprocessed and does not require any further processing. However, participants are free to apply any further processing techniques, if required.

Note

The dataset is imbalanced and participants need to consider this fact while training the model so as to give satisfactory performance on the test set. Also, subject level variability would play an important during performance of model on test set and hence should be addressed during the training of the model. For example, training and validation splits should be done on subject level instead of image level. In case this is not addressed, model may lead to poor performance on prospective subjects data.

Organizing Team

• Dr. Anubha Gupta, Associate Professor, SBILab, Deptt. of ECE, IIIT-Delhi, India,
  Lab: http://sbilab.iiitd.edu.in/index.html; email: anubha@iiitd.ac.in
• Dr. Ritu Gupta, MD, Professor and Laboratory Head, Dr. BRAIRCH, All India Institute of Medical Sciences (AIIMS), New Delhi, India;
  email: drritugupta@gmail.com
• Shiv Gehlot, PhD Scholar, SBILab, IIIT-Delhi, India; email: shivg@iiitd.ac.in
• Simmi Mourya, Research Assistant, SBILab, IIIT-Delhi, India; email: simmim@iiitd.ac.in

Contact Person

• Anubha Gupta (anubha@iiitd.ac.in)
• Shiv Gehlot (shivg@iiitd.ac.in)

The challenge workshop will be held at Hilton Molino Stucky, Venice, Italy from 2.45 PM-6.00 PM.  Please register yourself for the challenge day. 

The instructions to make the submissions are as follows:

The top entries of this challenge are: