Competition

Evaluation Metrics

Predicted contours are evaluated against the ground truth using:

• The overlap Dice metric (DM), based on the pixel labeling as the result of a segmentation algorithm, defined as 2*intersection of automatic and manual areas/(sum of automatic and manual areas);
• The Hausdorff distance (HD), defined as max(ha,hb), where ha is the maximum distance, for all automatic contour points, to the closest manual contour point and hb is the maximum distance, for all manual contour points, to the closest automatic contour point. The Hausdorff distance is computed in mm thanks to spatial resolution.

DM and HD are complementary metrics and provide a good idea of the global accuracy of a segmentation method. They will be computed independently for each of the 4 organs at risk. Evaluation code will be open source and easily accessible since these metrics are standard and already implemented in numerous libraries. We will use the library SimpleITK in our evaluation code. To this regard, 8 metrics will be computed and each one will be assigned to a designated row. The average of each of the 8 rows will give the challenger’s ranking. Moreover, as we wish to pay particular attention to esophagus segmentation, the leaderboard will display the average row metrics calculated for the esophagus. While it will not be taken into account in the final score, computational complexity and running time will be a point of discussion, in the papers, but also on the day of the challenge at ISBI.

Preparing your submission

The results must be submitted through this CodaLab competition site in the section Participate. The participants can make up to 3 submissions per day.

A result.zip for this competition would look like this:

result.zip 
    |- Patient_01.nii (Contains: segmented CT scan in nifty format)
    .
    .
    .
    |- Patient_n.nii

Warning: make sur to zip just the contents of the directory result not the directory itself

Please be patient. Once you've submitted your results, it may take a while for the scoring program to compute your scores.

Register to the SegTHOR challenge

To register to the SegTHOR challenge, please click on the "Participate" tab, and tick "Register".

You will have **direct** access to the data (no need to wait for the organizers approval). 

⚠️ You are not allowed to redistribute the data. By registering, you implicitely agree upon the data usage agreement, that can be downloaded here.

 Please mention the following reference to the SegTHOR database in any scientific presentation or reports (oral, or written, or posted on any Website or other publicly accessible online repository):

Lambert, Z., Petitjean, C., Dubray, B., & Kuan, S. (2020, November). SegTHOR: Segmentation of thoracic organs at risk in CT images. In 2020 Tenth International Conference on Image Processing Theory, Tools and Applications (IPTA) (pp. 1-6). IEEE.

 For any inquiry, please send an email to caroline.petitjean AT univ-rouen.fr

Data Quality

Computed Tomography images (with or without IV contrast) have been retrieved from the medical records of 60 patients with non small cell lung cancer referred for curative-ntent radiotherapy at the Centre Henri Becquerel, Rouen, France (CHB, regional anti-cancer center) between February 2016 and June 2017. Patients with tumor extension distorting the mediastinum anatomy were not eligible. All data were fully anonymized. The institution (CHB) board reviewed and approved the protocol in February 2016.

The CT scans have 512 x 512 pixels size with in-plane resolution varying between 0.90 mm and 1.37 mm per pixel, depending on the patient. The number of slices varies from 150 to 284 with a z-resolution between 2mm and 3.7mm. The most frequent resolution is 0.98x0.98x2.5 mm³.

Training & Testing Data

The whole SegTHOR dataset (60 patients and 11084 slices) has been randomly split into:

• a training set: 40 patients, 7390 slices
• a testing set: 20 patients, 3694 slices

We have carefully checked the distribution to ensure that the training is representative of the testing data. No specific preprocessing will be made. Participants are fully allowed to use their own training data, in addition to the ones provided by the challenge. As described in the general organization of the challenge in section Overwiew we plan to release the training data and their associated manual contours in early January, and the testing data by end-February (results submission will be expected by end-March).

Reference Standard

The reference standard for our problem is manual segmentation. On each CT scan, the OARs have been delineated by Pr Bernard Dubray, an experienced radiation oncologist, using a SomaVision platform, Varian Medical Systems, Inc, Palo Alto, USA. The body and lung contours were segmented with the automatic tools available on the platform. The esophagus was manually delineated from the 4th cervical vertebra to the esophago-gastric junction. The heart was delineated as recommended by the Radiation Therapy Oncology Group 2. The trachea was contoured from the lower limit of the larynx to 2cm below the carena excluding the lobar bronchi. The aorta was delineated from its origin above the heart down to below the diaphragm pillars.

Organizing Team

• Caroline Petitjean, PhD, LITIS EA 4108, Associate Professor at the Université de Rouen, France

• Zoé Lambert, Research Engineer, LMI, INSA de Rouen, France

• Pr. Su Ruan, PhD, LITIS EA 4108, Université de Rouen

• Pr. Bernard Dubray, MD, PhD, LITIS EA 4108, Université de Rouen, Radiation oncologist at the Centre Henri Becquerel, Rouen, France (regional anti-cancer center)

Contact Person

• Caroline Petitjean (caroline.petitjean@univ-rouen.fr)

• Zoé Lambert (zoe.lambert@insa-rouen.fr)

ISBI challenge

The proceedings of the SegTHOR challenge at IEEE ISBI 2019 have been published as Vol 2349 of CEUR-WS and may be found here.

ISBI Challenge Participants

The following participants presented their work during the SegTHOR challenge at IEEE ISBI in Venice Italy:

ISBI Challente Dates