SegTHOR

Organized by Zoe - Current server time: March 22, 2019, 6:08 p.m. UTC

Previous

Phase 1
Jan. 5, 2019, midnight UTC

Current

Phase 2
Feb. 28, 2019, midnight UTC

End

Competition Ends
April 8, 2019, midnight UTC

SegTHOR: Segmentation of THoracic Organs at Risk in CT images

News!! March 14: submission system is now open, please follow this link

Our challenge addresses the problem of organs at risk segmentation in Computed Tomography (CT) images. In lung and esophageal cancer, radiation therapy is a treatment of choice, and the irradiation planning begins with the delineation of the target tumor and healthy organs located near the target tumor, called Organs at Risk (OAR) on CT images. Routinely, the delineation is largely manual which is tedious and may be source of reproducibility errors. For some organs (e.g. esophagus), the segmentation is especially challenging: shape and position vary greatly between patients; the contours in CT images have low contrast, and can be absent. In this challenge, we focus of 4 OAR: heart, aorta, trachea, esophagus, which are shown on the figure below.

 

img

 

The goal of the SegTHOR challenge is to automatically segment 4 OAR: heart, aorta, trachea, esophagus. Participants will be provided with a training set 40 CT scans with manual segmentation. The test set will include 20 CT scans.

Warning: before registration don't forget to complete the form available within the terms and conditions section.

 

Important Dates

Date Description
05-Jan, 2019 Release of the training set
28-Feb, 2019 Release of the test set 
21-Mar, 2019 Deadline for the paper submission that include best results on the test set
08-Apr, 2019

Workshop in Venice, Italy at IEEE ISBI conference  
at the Hilton Molino Stucky, Venetian Ballroom E, from 9:00 til 12:15

 

 

 

 

 

 

 

Publications

After the challenge, we plan to write a paper describing the data, the methods and summarizing the results. It will be submitted to a relevant journal (Medical Image Analysisor IEEE Transactions on Medical Imaging). Proceedings will also be made available on the challenge website or hosting framework. There is no previous version of the challenge as this is the first time this challenge will take place. Similarly, the dataset has never been released publicly, yet, and will be revealed for the first time. Since we do not have yet a paper describing the dataset, but have been working on this dataset developing our own segmentation method, we will ask the teams using our dataset to sign adata usage agreement form and to acknowledge the SegTHOR dataset by citing the following paper:

Roger Trullo, C. Petitjean, Su Ruan, Bernard Dubray, Dong Nie, and Dinggang Shen.
Segmentation of organs at risk in thoracic CT images using a sharpmask architecture and
conditional random fields. In 14th IEEE International Symposium on Biomedical Imaging
(ISBI), pp. 1003-1006, 2017.


In the future, this acknowledgement to the SegTHOR dataset via this paper will be updated with a paper only describing the data, when published, and later, the paper summarizing the challenge results. Teams exceeding minimum performance requirements will be invited to participate as co-authors.

Evaluation Metrics

We will use standard evaluation metrics when comparing contours, namely:

  • The overlap Dice metric (DM), based on the pixel labeling as the result of a segmentation algorithm, defined as 2*intersection of automatic and manual areas/(sum of automatic and manual areas);
  • The Hausdorff distance (HD), defined as max(ha,hb), where ha is the maximum distance, for all automatic contour points, to the closest manual contour point and hb is the maximum distance, for all manual contour points, to the closest automatic contour point. The Hausdorff distance is computed in mm thanks to spatial resolution.

DM and HD are complementary metrics and provide a good idea of the global accuracy of a segmentation method. Evaluation code will be open source and easily accessible since these metrics are standard and already implemented in numerous libraries. We will use the SimpleITK library in our evaluation code. DM et HD will be computed independently for each of the 4 organs at risk.

Scoring

Since DM et HD will be computed independently for each of the 4 organs at risk8 measures will be obtained. For each measure, the challenger is assigned a rank. The average of each of the 8 ranks will give the challenger’s ranking. Moreover, as we wish to pay particular attention to esophagus segmentation, the leaderboard will display the average measures calculated for the esophagus. While it will not be taken into account in the final score, computational complexity and running time will be a point of discussion, in the papers, but also on the day of the challenge at ISBI.

Preparing your submission

The results must be submitted through this CodaLab competition site in the section Participate. The participants can make up to 3 submissions per day. Moreover, we will ask to the participants to provide a paper describing their method.

A result.zip for this competition would look like this:

result.zip 
    |- Patient_01.nii (Contains: segmented CT scan in nifty format)
.
.
.
|- Patient_n.nii

Warning: make sur to zip just the contents of the directory result not the directory itself

Please be patient. Once you've submitted your results, it may take a while for the scoring program to compute your scores.

Paper submission

Papers must follow the ISBI format. That is to say, the IEEE format with double column and maximum 4 pages. You can check examples directly on the ISBI website. After the challenge, we will create proceedings of our workshop that will be stored on arXiv and on our website.

Data Quality

Computed Tomography images (with or without IV contrast) have been retrieved from the medical records of 60 patients with non small cell lung cancer referred for curative-ntent radiotherapy at the Centre Henri Becquerel, Rouen, France (CHB, regional anti-cancer center) between February 2016 and June 2017. Patients with tumor extension distorting the mediastinum anatomy were not eligible. All data were fully anonymized. The institution (CHB) board reviewed and approved the protocol in February 2016.


The CT scans have 512 x 512 pixels size with in-plane resolution varying between 0.90 mm and 1.37 mm per pixel, depending on the patient. The number of slices varies from 150 to 284 with a z-resolution between 2mm and 3.7mm. The most frequent resolution is 0.98x0.98x2.5 mm3.

Training & Testing Data

The whole SegTHOR dataset (60 patients and 11084 slices) has been randomly split into:

  • a training set: 40 patients, 7390 slices
  • a testing set: 20 patients, 3694 slices

We have carefully checked the distribution to ensure that the training is representative of the testing data. No specific preprocessing will be made. Participants are fully allowed to use their own training data, in addition to the ones provided by the challenge. As described in the general organization of the challenge in section Overwiew we plan to release the training data and their associated manual contours in early January, and the testing data by end-February (results submission will be expected by end-March).

Reference Standard

The reference standard for our problem is manual segmentation. On each CT scan, the OARs have been delineated by Pr Bernard Dubray, an experienced radiation oncologist, using a SomaVision platform, Varian Medical Systems, Inc, Palo Alto, USA. The body and lung contours were segmented with the automatic tools available on the platform. The esophagus was manually delineated from the 4th cervical vertebra to the esophago-gastric junction. The heart was delineated as recommended by the Radiation Therapy Oncology Group 2. The trachea was contoured from the lower limit of the larynx to 2cm below the carena excluding the lobar bronchi. The aorta was delineated from its origin above the heart down to below the diaphragm pillars.

 

Terms and Conditions

In order to participate to this competition, it is mandatory to sign up and fill in the above form. The form should be returned to us completed and signed for you to be allowed in this competition.

You can find the form here.

Organizing Team

  • Pr. Bernard Dubray, MD, PhD, LITIS EA 4108, Université de Rouen, Radiation oncologist at the Centre Henri Becquerel, Rouen, France (regional anti-cancer center)

Contact Person

  • Caroline Petitjean (caroline.petitjean@univ-rouen.fr)
  • Zoé Lambert (zoe.lambert@insa-rouen.fr)

Paper submission

In order to participate to SegTHOR challenge to be held during IEEE ISBI'19 in Venice Italy, please submit your paper here.

Please use the IEEE ISBI submission template.

Articles should be between 2-pages and 4-pages long.

Notification of acceptance will be released on March 28th, 2019 at the latest.

Let us know if you have any problem with the submission system.

Deadline: March 21st, 2019, 23:59 UTC+01:00

Phase 1

Start: Jan. 5, 2019, midnight

Description: Training phase (with optional result scoring)

Phase 2

Start: Feb. 28, 2019, midnight

Description: Test phase with result scoring

Competition Ends

April 8, 2019, midnight

You must be logged in to participate in competitions.

Sign In