eHealth KD @TASS18

Organized by tass18-task3 - Current server time: Oct. 16, 2018, 8:38 p.m. UTC
Reward $100

Previous

Test
May 7, 2018, midnight UTC

Current

Train
April 2, 2018, midnight UTC

End

Competition Ends
May 28, 2018, 11:59 p.m. UTC

Natural Language Processing (NLP) methods are increasingly being used to mine knowledge from unstructured health texts. Recent advances in health text processing techniques are encouraging researchers and medical domain experts to go beyond just reading the information included in published texts (e.g. academic manuscripts, clinical reports, etc.) and structured questionnaires, to discover new knowledge by mining health contents. This has allowed other perspectives to surface that were not previously available.

Over the years many eHealth challenges have taken place, which have attempted to identify, classify, extract and link knowledge, such as Semevals, CLEF campaigns and others.

Inspired by previous NLP shared tasks like “ Semeval-2017 Task 10: ScienceIE” and research lines like Teleologies, both not specifically focussed on the health area, eHealth-KD proposes modelling the human language in a scenario in which Spanish electronic health documents could be machine readable from a semantic point of view. With this task, we expect to encourage the development of software technologies to automatically extract a large variety of knowledge from eHealth documents written in the Spanish Language.

The documents used as corpus have been taken from MedlinePlus and manually processed to make them fit for the task. Additional details are provided at the end of this document.

To achieve this purpose, three subtasks are presented:

Subtask A: Identification of keyphrases 
Subtask B: Classification of key phrases 
Subtask C: Setting semantic relationships

There will be three evaluation scenarios:

Scenario 1: Only plain text is given (Subtasks A, B, C).

In this first scenario, the participants will perform the three subtasks consecutively and provide the corresponding development output files. The only input provided are plain text files input_<topic>.txt for a particular list of topics that were not released with the training data.

Systems will be ranked according to an aggregated F1 metric computed on the three tasks, by considering precision and recall as follows:

Besides this aggregated F1 score, individual F1 scores for each of the subtasks will also be reported.

Scenario 2: Plain text and manually annotated key phrase boundaries are given (Subtasks B, C).

In this second scenario participants will perform tasks B and C sequentially, and provide the corresponding output files. As input, they receive both plain text files ( input_<topic>.txt ), and the corresponding gold files for the task A ( output_A_<topic>.txt ). The purpose of this scenario is to evaluate the quality of tasks B and C independently from task A. As in the previous scenario, an aggregated F1 metric is reported, based on the following precision and recall :

Besides the aggregated F1 metric, individual scores of F1 for each of the subtasks are also reported.

Scenario 3: Plain text with manually annotated key phrases and their types are given (Subtask C).

In this scenario both the gold outputs for task A and task B are provided, and the participants must only perform the process to obtain task C output files. The purpose of this scenario is to evaluate only the quality of task C independently of the complexity of task A and B. As before, an aggregated F1 metric is reported, based on the following precision and recall :

By submitting results to this competition, you consent to the public release of your scores at the TASS-2018 workshop and in the associated proceedings, at the task organizers' discretion. Scores may include, but are not limited to, automatic and manual quantitative judgements, qualitative judgements, and such other metrics as the task organizers see fit. You accept that the ultimate decision of metric choice and score value is that of the task organizers.

You further agree that the task organizers are under no obligation to release scores and that scores may be withheld if it is the task organizers' judgement that the submission was incomplete, erroneous, deceptive, or violated the letter or spirit of the competition's rules. Inclusion of a submission's scores is not an endorsement of a team or individual's submission, system, or science.

You further agree that your system may be named according to the team name provided at the time of submission, or to a suitable shorthand as determined by the task organizers.

You agree not to redistribute the test data except in the manner prescribed by its licence.

Trial

Start: Feb. 12, 2018, midnight

Train

Start: April 2, 2018, midnight

Test

Start: May 7, 2018, midnight

Competition Ends

May 28, 2018, 11:59 p.m.

You must be logged in to participate in competitions.

Sign In