SegTHOR

Organized by caroline.petitjean - Current server time: Nov. 17, 2019, 10:25 a.m. UTC

Previous

Phase 1
Jan. 5, 2019, midnight UTC

Current

Phase 2
Feb. 28, 2019, midnight UTC

End

Competition Ends
Never

SegTHOR: Segmentation of THoracic Organs at Risk in CT images

News! Sept 4, 2019: following Codalab's crash, SegTHOR competition is back  

Our challenge was an official challenge of IEEE ISBI in April 2019 and remains open for new submissions.

Our challenge addresses the problem of organs at risk segmentation in Computed Tomography (CT) images. In lung and esophageal cancer, radiation therapy is a treatment of choice, and the irradiation planning begins with the delineation of the target tumor and healthy organs located near the target tumor, called Organs at Risk (OAR) on CT images. Routinely, the delineation is largely manual which is tedious and may be source of reproducibility errors. For some organs (e.g. esophagus), the segmentation is especially challenging: shape and position vary greatly between patients; the contours in CT images have low contrast, and can be absent. In this challenge, we focus of 4 OAR: heart, aorta, trachea, esophagus, which are shown on the figure below.

 

img

 

 

The goal of the SegTHOR challenge is to automatically segment 4 OAR: heart, aorta, trachea, esophagus. We provide a training set 40 CT scans with manual segmentation. The test set includes 20 CT scans.

Publications

September 2019: We are now writing a paper describing the data, the submitted methods and summarizing the results.

We ask the teams using our dataset to sign adata usage agreement form and to acknowledge the SegTHOR dataset by citing the following paper:

R Trullo, C Petitjean, B Dubray, S Ruan. Multiorgan segmentation using distance-aware adversarial networks, Journal of Medical Imaging 6 (1), 014001, 2019

Evaluation Metrics

Predicted contours are evaluated against the ground truth using:

  • The overlap Dice metric (DM), based on the pixel labeling as the result of a segmentation algorithm, defined as 2*intersection of automatic and manual areas/(sum of automatic and manual areas);
  • The Hausdorff distance (HD), defined as max(ha,hb), where ha is the maximum distance, for all automatic contour points, to the closest manual contour point and hb is the maximum distance, for all manual contour points, to the closest automatic contour point. The Hausdorff distance is computed in mm thanks to spatial resolution.

DM and HD are complementary metrics and provide a good idea of the global accuracy of a segmentation method. They will be computed independently for each of the 4 organs at risk. Evaluation code will be open source and easily accessible since these metrics are standard and already implemented in numerous libraries. We will use the library SimpleITK in our evaluation code. To this regard, 8 metrics will be computed and each one will be assigned to a designated row. The average of each of the 8 rows will give the challenger’s ranking. Moreover, as we wish to pay particular attention to esophagus segmentation, the leaderboard will display the average row metrics calculated for the esophagus. While it will not be taken into account in the final score, computational complexity and running time will be a point of discussion, in the papers, but also on the day of the challenge at ISBI.

Preparing your submission

The results must be submitted through this CodaLab competition site in the section Participate. The participants can make up to 3 submissions per day.

A result.zip for this competition would look like this:

result.zip 
    |- Patient_01.nii (Contains: segmented CT scan in nifty format)
.
.
.
|- Patient_n.nii

Warning: make sur to zip just the contents of the directory result not the directory itself

Please be patient. Once you've submitted your results, it may take a while for the scoring program to compute your scores.

Terms and Conditions

In order to participate to this competition, it is mandatory to sign up and fill in a data usage agreement. The form should be returned to us completed and signed for you to be allowed in this competition.

Download the form here.

Fill it, sign it and return it to caroline.petitjean AT univ-rouen.fr

Data Quality

Computed Tomography images (with or without IV contrast) have been retrieved from the medical records of 60 patients with non small cell lung cancer referred for curative-ntent radiotherapy at the Centre Henri Becquerel, Rouen, France (CHB, regional anti-cancer center) between February 2016 and June 2017. Patients with tumor extension distorting the mediastinum anatomy were not eligible. All data were fully anonymized. The institution (CHB) board reviewed and approved the protocol in February 2016.


The CT scans have 512 x 512 pixels size with in-plane resolution varying between 0.90 mm and 1.37 mm per pixel, depending on the patient. The number of slices varies from 150 to 284 with a z-resolution between 2mm and 3.7mm. The most frequent resolution is 0.98x0.98x2.5 mm3.

Training & Testing Data

The whole SegTHOR dataset (60 patients and 11084 slices) has been randomly split into:

  • a training set: 40 patients, 7390 slices
  • a testing set: 20 patients, 3694 slices

We have carefully checked the distribution to ensure that the training is representative of the testing data. No specific preprocessing will be made. Participants are fully allowed to use their own training data, in addition to the ones provided by the challenge. As described in the general organization of the challenge in section Overwiew we plan to release the training data and their associated manual contours in early January, and the testing data by end-February (results submission will be expected by end-March).

Reference Standard

The reference standard for our problem is manual segmentation. On each CT scan, the OARs have been delineated by Pr Bernard Dubray, an experienced radiation oncologist, using a SomaVision platform, Varian Medical Systems, Inc, Palo Alto, USA. The body and lung contours were segmented with the automatic tools available on the platform. The esophagus was manually delineated from the 4th cervical vertebra to the esophago-gastric junction. The heart was delineated as recommended by the Radiation Therapy Oncology Group 2. The trachea was contoured from the lower limit of the larynx to 2cm below the carena excluding the lobar bronchi. The aorta was delineated from its origin above the heart down to below the diaphragm pillars.

 

Organizing Team

  • Pr. Bernard Dubray, MD, PhD, LITIS EA 4108, Université de Rouen, Radiation oncologist at the Centre Henri Becquerel, Rouen, France (regional anti-cancer center)

Contact Person

  • Caroline Petitjean (caroline.petitjean@univ-rouen.fr)
  • Zoé Lambert (zoe.lambert@insa-rouen.fr)

ISBI challenge

The proceedings of the SegTHOR challenge at IEEE ISBI 2019 have been published as Vol 2349 of CEUR-WS and may be found here.

ISBI Challenge Participants

The following participants presented their work during the SegTHOR challenge at IEEE ISBI in Venice Italy:

Name Institution Country Rank
S. Kim et al Yonsei University, Yonsei University College of Medicine South Korea 34
V. Kondratenko et al Skolkovo Institute of Science and Technology Russia 26
D. Lachinov Intel, Nizhny Novgorod Russia 15
L. van Harten et al Image Sciences Institute & Department of Radiotherapy, UMC Utrecht The Netherlands 12
S. Vesal et al Friedrich-Alexander-Universit\"at Erlangen-N\"urnberg Germany 9
Q. Wang et al Chinese University of HK, Sun Yat-sen University, University of Hong Kong HK, China 5
M. Han et al Shanghai United Imaging Intelligence Inc. China 1

 

 

 

 

 

 

ISBI Challente Dates

Date Description
05-Jan, 2019 Release of the training set
28-Feb, 2019 Release of the test set 
30-Mar, 2019 Last date for submission of results on the test set with a detailed paper 
08-Apr, 2019 Workshop in Venise, Italy at IEEE International Symposium on Biomedical Imaging (ISBI) conference  

Phase 1

Start: Jan. 5, 2019, midnight

Description: Training phase (with optional result scoring)

Phase 2

Start: Feb. 28, 2019, midnight

Description: Test phase with result scoring

Competition Ends

Never

You must be logged in to participate in competitions.

Sign In