SegTHOR

Organized by Zoe - Current server time: May 23, 2019, 6:46 p.m. UTC

Previous

Phase 1
Jan. 5, 2019, midnight UTC

Current

Phase 2
April 10, 2019, midnight UTC

End

Competition Ends
Never

SegTHOR: Segmentation of THoracic Organs at Risk in CT images


News!! April 25: SegTHOR proceedings are now online at http://ceur-ws.org/Vol-2349/

News!! April 15: The SegTHOR challenge took place at IEEE ISBI'19. The leaderboard was frozen to establish ranking of the participants. Now the leaderboard is open again, you can still download training and test set, and submit your results!

To see the challenge selected papers, results and photos, please see "Challenge results at ISBI" on the left panel.

News!! April 7: the challenge schedule at ISBI is now available, please follow this link

News!! March 14: submission system is now open, please follow this link


Our challenge addresses the problem of organs at risk segmentation in Computed Tomography (CT) images. In lung and esophageal cancer, radiation therapy is a treatment of choice, and the irradiation planning begins with the delineation of the target tumor and healthy organs located near the target tumor, called Organs at Risk (OAR) on CT images. Routinely, the delineation is largely manual which is tedious and may be source of reproducibility errors. For some organs (e.g. esophagus), the segmentation is especially challenging: shape and position vary greatly between patients; the contours in CT images have low contrast, and can be absent. In this challenge, we focus of 4 OAR: heart, aorta, trachea, esophagus, which are shown on the figure below.

 

img

 

The goal of the SegTHOR challenge is to automatically segment 4 OAR: heart, aorta, trachea, esophagus. Participants will be provided with a training set 40 CT scans with manual segmentation. The test set will include 20 CT scans.

Warning: before registration don't forget to complete the form available within the terms and conditions section.

 

Important Dates

Date Description
05-Jan, 2019 Release of the training set
28-Feb, 2019 Release of the test set 
21-Mar, 2019 Deadline for the paper submission that include best results on the test set
08-Apr, 2019

Workshop in Venice, Italy at IEEE ISBI conference  

 

 

 

 

 

 

 

Publications

After the challenge, we plan to write a paper describing the data, the methods and summarizing the results. It will be submitted to a relevant journal (Medical Image Analysisor IEEE Transactions on Medical Imaging). Proceedings will also be made available on the challenge website or hosting framework. There is no previous version of the challenge as this is the first time this challenge will take place. Similarly, the dataset has never been released publicly, yet, and will be revealed for the first time. Since we do not have yet a paper describing the dataset, but have been working on this dataset developing our own segmentation method, we will ask the teams using our dataset to sign adata usage agreement form and to acknowledge the SegTHOR dataset by citing the following paper:

Roger Trullo, C. Petitjean, Su Ruan, Bernard Dubray, Dong Nie, and Dinggang Shen.
Segmentation of organs at risk in thoracic CT images using a sharpmask architecture and
conditional random fields. In 14th IEEE International Symposium on Biomedical Imaging
(ISBI), pp. 1003-1006, 2017.


In the future, this acknowledgement to the SegTHOR dataset via this paper will be updated with a paper only describing the data, when published, and later, the paper summarizing the challenge results. Teams exceeding minimum performance requirements will be invited to participate as co-authors.

Evaluation Metrics

We will use standard evaluation metrics when comparing contours, namely:

  • The overlap Dice metric (DM), based on the pixel labeling as the result of a segmentation algorithm, defined as 2*intersection of automatic and manual areas/(sum of automatic and manual areas);
  • The Hausdorff distance (HD), defined as max(ha,hb), where ha is the maximum distance, for all automatic contour points, to the closest manual contour point and hb is the maximum distance, for all manual contour points, to the closest automatic contour point. The Hausdorff distance is computed in mm thanks to spatial resolution.

DM and HD are complementary metrics and provide a good idea of the global accuracy of a segmentation method. Evaluation code will be open source and easily accessible since these metrics are standard and already implemented in numerous libraries. We will use the SimpleITK library in our evaluation code. DM et HD will be computed independently for each of the 4 organs at risk.

Scoring

Since DM et HD will be computed independently for each of the 4 organs at risk8 measures will be obtained. For each measure, the challenger is assigned a rank. The average of each of the 8 ranks will give the challenger’s ranking. Moreover, as we wish to pay particular attention to esophagus segmentation, the leaderboard will display the average measures calculated for the esophagus. While it will not be taken into account in the final score, computational complexity and running time will be a point of discussion, in the papers, but also on the day of the challenge at ISBI.

Preparing your submission

The results must be submitted through this CodaLab competition site in the section Participate. The participants can make up to 3 submissions per day. Moreover, we will ask to the participants to provide a paper describing their method.

A result.zip for this competition would look like this:

result.zip 
    |- Patient_01.nii (Contains: segmented CT scan in nifty format)
.
.
.
|- Patient_n.nii

Warning: make sur to zip just the contents of the directory result not the directory itself

Please be patient. Once you've submitted your results, it may take a while for the scoring program to compute your scores.

Paper submission

Papers must follow the ISBI format. That is to say, the IEEE format with double column and maximum 4 pages. You can check examples directly on the ISBI website. After the challenge, we will create proceedings of our workshop that will be stored on arXiv and on our website.

Data Quality

Computed Tomography images (with or without IV contrast) have been retrieved from the medical records of 60 patients with non small cell lung cancer referred for curative-ntent radiotherapy at the Centre Henri Becquerel, Rouen, France (CHB, regional anti-cancer center) between February 2016 and June 2017. Patients with tumor extension distorting the mediastinum anatomy were not eligible. All data were fully anonymized. The institution (CHB) board reviewed and approved the protocol in February 2016.


The CT scans have 512 x 512 pixels size with in-plane resolution varying between 0.90 mm and 1.37 mm per pixel, depending on the patient. The number of slices varies from 150 to 284 with a z-resolution between 2mm and 3.7mm. The most frequent resolution is 0.98x0.98x2.5 mm3.

Training & Testing Data

The whole SegTHOR dataset (60 patients and 11084 slices) has been randomly split into:

  • a training set: 40 patients, 7390 slices
  • a testing set: 20 patients, 3694 slices

We have carefully checked the distribution to ensure that the training is representative of the testing data. No specific preprocessing will be made. Participants are fully allowed to use their own training data, in addition to the ones provided by the challenge. As described in the general organization of the challenge in section Overwiew we plan to release the training data and their associated manual contours in early January, and the testing data by end-February (results submission will be expected by end-March).

Reference Standard

The reference standard for our problem is manual segmentation. On each CT scan, the OARs have been delineated by Pr Bernard Dubray, an experienced radiation oncologist, using a SomaVision platform, Varian Medical Systems, Inc, Palo Alto, USA. The body and lung contours were segmented with the automatic tools available on the platform. The esophagus was manually delineated from the 4th cervical vertebra to the esophago-gastric junction. The heart was delineated as recommended by the Radiation Therapy Oncology Group 2. The trachea was contoured from the lower limit of the larynx to 2cm below the carena excluding the lobar bronchi. The aorta was delineated from its origin above the heart down to below the diaphragm pillars.

 

Terms and Conditions

In order to participate to this competition, it is mandatory to sign up and fill in the above form. The form should be returned to us completed and signed for you to be allowed in this competition.

You can find the form here.

Organizing Team

  • Pr. Bernard Dubray, MD, PhD, LITIS EA 4108, Université de Rouen, Radiation oncologist at the Centre Henri Becquerel, Rouen, France (regional anti-cancer center)

Contact Person

  • Caroline Petitjean (caroline.petitjean@univ-rouen.fr)
  • Zoé Lambert (zoe.lambert@insa-rouen.fr)

The organizers thank the M2SiNum project (co-financed by the European Union via the European Regional Development Fund and by the Normandie Regional Council) for supporting this work.

In early January 2019, the SegTHOR challenge was launched. Test set was released at the end of February, and challengers had 3 weeks and 10 attemps to submit their best label maps obtained on the test Set.

We received 50 submissions on the Test Set from the following countries:

On March 21, we had received 12 paper submissions summarizing the methods and the results obtained on this dataset.

All papers submitted and accepted during the challenge can be downloaded all at once herePublishing proper proceedings is ongoing.

On April 8th 2019, the challenge took place, with 7 teams on site to present their work. The schedule was as follows:

09:00 Caroline Petitjean (LITIS, University of Rouen France)
  Presentation of the SegTHOR challenge 
   
09:25 Sekeun Kim, Yeonggul Jang, Kyunghoon Han, Hackjoon Shim and Hyuk-Jae Chang (Yonsei University, Yonsei University College of Medicine, South Korea)
  A Cascaded Two-step Approach For Segmentation of Thoracic Organs (paper #3)
   
09:40 Vladimir Kondratenko, Dmitry Denisenko, Artem Pimkin and Mikhail Belyaev (Skolkovo Institute of Science and Technology, Russia)
  Segmentation of thoracic organs at risk in ct images using Localization and organ-specific cnn (paper #9)
   
09:55 Dmitry Lachinov ( Intel, Nizhny Novgorod, Russia)
  Segmentation of Thoracic Organs Using Pixel Shuffle (paper #10)
   
10:10 Louis van Harten, Julia Noothout, Joost Verhoeff, Jelmer Wolterink and Ivana Išgum (Image Sciences Institute, UMC Utrecht Department of Radiotherapy, UMC Utrecht, The Netherlands)
  Automatic Segmentation of Organs at Risk in Thoracic CT scans by Combining 2D and 3D Convolutional Neural Networks (paper #12)
   
11:00 Sulaiman Vesal, Nishant Ravikumar and Andreas Maier (Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany
  A 2D dilated residual U-Net for multi-organ segmentation in thoracic CT (paper #13)
   
11:15 Qin Wang, Weibing Zhao, Chunhui Zhang, Zhen Li, Shuguang Cui, Guanbin Li, Liyue Zhang and Changmiao Wang (Chinese University of HK, Sun Yat-sen University, The University of Hong Kong, China)
  3D Enhanced multi-scale network for thoracic organs segmentation (paper #8)
   
11:30 Miaofei Han, Guang Yao, Wenhai Zhang, Guangrui Mu, Yiqiang Zhan, Xiang Zhou and Yaozong Gao (Shanghai United Imaging Intelligence Inc., China)
  Segmentation of CT thoracic organs by multi-resolution VB-nets (paper #1)
   
11:45 Results and discussion (Caroline Petitjean, LITIS, University of Rouen France)

 

Acknowledgements: to ISBI challenge chairs Tom Vercauteren and Ivana Isgum, and to all the teams that had their work presented during ISBI, ie Kyunghoon Han, Sekeun Kim, Julia Noothout, Guang Yao, Jelmer Wolterink, Hackjoon Shim, Miaofei Han, Guangrui Mu, Xiang Zhou, Ivana Išgum, Guanbin Li, Wenhai Zhang, Dmitry Lachinov, Changmiao Wang, Hyuk-Jae Chang, Zhen Li, Liyue Zhang, Qin Wang, Shuguang Cui, Nishant Ravikumar, Vladimir Kondratenko, Yiqiang Zhan, Dmitry Denisenko, Weibing Zhao, Joost Verhoeff, Artem Pimkin, Andreas Maier, Mikhail Belyaev, Chunhui Zhang, Sulaiman Vesal, Louis van Harten, Yaozong Gao, Yeonggul Jang.

Thank you to Minh Ha Van NGUYEN who helped to produce the results.

 We are now working towards the paper that will summarizes all the results. Stay tuned! ;-)

A few photos of the event:

 

All papers accepted at the challenge can be downloaded all at once here.

SegTHOR proceedings are available at: http://ceur-ws.org/Vol-2349/

NB: in bold, the teams that were present at the ISBI challenge.

Miaofei Han, Guang Yao, Wenhai Zhang, Guangrui Mu, Yiqiang Zhan, Xiang Zhou and Yaozong Gao
Segmentation of CT thoracic organs by multi-resolution VB-nets rank #1

Tao He, Jixiang Guo, Jianyong Wang, Xiuyuan Xu and Yi Zhang
Multi-task Learning for the Segmentation of Thoracic Organs at Risk in CT images rank #2

Sekeun Kim, Yeonggul Jang, Kyunghoon Han, Hackjoon Shim and Hyuk-Jae Chang
A Cascaded Two-step Approach For Segmentation of Thoracic Organs rank #34

Pan Chen, Chenghai Xu, Fenglong Sun, Xiaoying Li and Yingying Ma
Two-stage Network for OAR segmentation rank #4

Li Zhang, Lishen Wang, Yijie Huang and Huai Chen
SEGMENTATION OF THORACIC ORGANS AT RISK IN CT IMAGES COMBINING COARSE AND FINE NETWORK rank #20

Manoj Satya Kumar Gali, Neerja Garg and Srikanth Vasamsetti
Dilated U-Net based Segmentation of Organs at Risk in Thoracic CT Images rank #46

Qin Wang, Weibing Zhao, Chunhui Zhang, Zhen Li, Shuguang Cui, Guanbin Li, Liyue Zhang and Changmiao Wang
3D ENHANCED MULTI-SCALE NETWORK FOR THORACIC ORGANS SEGMENTATION rank #5

Vladimir Kondratenko, Dmitry Denisenko, Artem Pimkin and Mikhail Belyaev
SEGTHOR: SEGMENTATION OF THORACIC ORGANS AT RISK IN CT IMAGES rank #26

Dmitry Lachinov
Segmentation of Thoracic Organs Using Pixel Shuffle rank #15

Ming Feng, Weiquan Huang, Yin Wang  and Yuxia Xie
Multi-Organ Segmentation Using Simplified Dense V-Net With Post Processing rank #19

Louis van Harten, Julia Noothout, Joost Verhoeff, Jelmer Wolterink and Ivana Išgum
Automatic Segmentation of Organs at Risk in Thoracic CT scans by Combining 2D and 3D Convolutional Neural Networks rank #12

Sulaiman Vesal, Nishant Ravikumar and Andreas Maier
A 2D dilated residual U-Net for multi-organ segmentation in thoracic CT rank #9

Phase 1

Start: Jan. 5, 2019, midnight

Description: Training phase (with optional result scoring)

Phase 2

Start: April 10, 2019, midnight

Description: Test phase with result scoring

Competition Ends

Never

You must be logged in to participate in competitions.

Sign In