Evaluation measures

The main goal of this challenge is to automatically segment liver lesions in CT volumes. We also evaluate liver segmentation and tumor burden estimation.

We evaluate detection and segmentation performance, separately. Segmentation metrics are evaluated only for detected lesions, except for a global Dice score (combines all data sets into one) and an average Dice per volume score. A lesion is considered detected by a contiguous predicted object which has a greater than 50% intersection over union with the reference lesion. Per-lesion segmentations metrics for detected lesions are the mean values for Dice score, as well as Jaccard and volume overlap error (VOE), relative volume difference (RVD), average symmetric surface distance (ASSD), and maximum symmetric surface distance (MSSD). Liver segmentation is evaluated with the same segmentation metrics. If any liver prediction is not provided, all average liver segmentation metrics will default to the worst possible score (0 or infinity, depending on the metric). The tumor burden estimation is evaluated as a a root mean square (RMS) and a max difference/error with respect to to the true tumour burden. Please have a look at our metrics notebook once it becomes available.

Publication

The top 10 participating teams and individuals will be invited to contribute to a joint journal paper describing and summarising the methods used and results found in this challenge. The paper will be submitted to a high-impact journal in the field no later than 6 months after the challenge. In order to be included in the joint journal paper the participants must send to the organisers or submit to arXiv a 4 page paper in appropriate LNCS latex describing their methods. The organisers will review the paper for sufficient detail to be able to understand and reproduce the method and hold the right to exclude participants from the joint journal paper in case their method description is not adequate.

Attribution

Appropriate citation is to be made in scientific publications (journal publications, conference papers, technical reports, presentations at conferences and meetings) that use the data shared in this challenge. Currently, this citation must refer to this website, and later to the publication that will describe the results of this challenge. Teams must notify the organisers of the challenge about any publication that is (partly) based on the results or data published on this site in order for us to maintain a list of publications associated with the challenge.

Data license

Creative Commons Licence This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License. Feel free to contact us if you have any questions or need clarification regarding the rules of the challenge or the licensing of the data.

Organizer and Data Contributors

Technical University of Munich

Patrick Christ, Florian Ettlinger, Felix Gruen, Sebastian Schlecht, Jana Lipkova, Georgios Kassis, Sebastian Ziegelmayer, Fabian Lohöfer, Rickmer Braren & Bjoern Menze

Ludwig Maxmilian University of Munich

Julian Holch, Felix Hofmann, Wieland Sommer & Volker Heinemann

Radboudumc

Colin Jacobs, Gabriel Efrain HumpireMamani & Bram van Ginneken

Polytechnique Montréal & CHUM Research Center

Gabriel Chartrand, Eugene Vorontsov, An Tang, Michal Drozdzal & Samuel Kadoury

Tel Aviv University & Sheba Medical Center

Avi Ben-Cohen, Eyal Klang, Marianne M. Amitai, Eli Konen & Hayit Greenspan.

IRCAD

Johan Moreau, Alexandre Hostettler & Luc Soler

The Hebrew University of Jerusalem & Hadassah University Medical Center

Refael Vivanti, Adi Szeskin, Naama Lev-Cohain, Jacob Sosna & Leo Joskowicz

Special thanks to the CodaLab Team for helping us

Eric Carmichael & Flavio Alexander

MICCAI 2017

The LITS Challenge Workshop will take place on 14.09.2017. We are happy to announce our Keynote Speaker Olaf Ronneberger (Deepmind).

ISBI 2017

The LITS Challenge Workshop at ISBI 2017 took place on 18.04.2017 in Melbourne, Australia. Please find attached to official leaderboard from ISBI 2017 LITS Challenge.

FAQ

If you have questions please have a look at our Google Group. If you encounter any issues during the submission process please have a look at the CodaLab Documentation.