Competition

Schemes

There will be three evaluation scenarios:

1. Only plain text is given (Subtasks A, B, C)

2. Plain text with manually annotated keyphrase boundaries are given (Subtasks B, C)

3. Plain text with manually annotated keyphrases and their types are given (Subtask C)

Metrics

The output of systems is matched exactly against the gold standard. The traditionally used metrics of precision, recall and F1-score are computed and the micro-average of those metrics across publications of the three genres are calculated. These metrics are calculated for Subtasks A, B and C.

Additional Resources

Participants may use additional external resources, as long as they declare this at submission time. However, participants may not manually annotate the test data.

There are three subtasks:

Subtask (A): Identification of keyphrases

Given a scientific publication, the goal of this task is to identify all the keyphrases in the document.

Subtask (B): Classification of identified keyphrases

In this task, each keyphrase needs to be labelled by one of three types: (i) PROCESS, (ii) TASK, and (iii) MATERIAL.

PROCESS

Keyphrases relating to some scientific model, algorithm or process should be labelled by PROCESS.

TASK

Keyphrases those denote the application, end goal, problem, task should be labelled by TASK.

MATERIAL

MATERIAL keyphrases identify the resources used in the paper.

Subtask (C): Extraction of relationships between two identified keyphrases

Every pair of keyphrases need to be labelled by one of three types: (i) HYPONYM-OF, (ii) SYNONYM-OF, and (iii) NONE.

HYPONYM-OF

The realtionship between two keyphrases A and B is HYPONYM-OF if semantic field of A is included within that of B. One example is Red HYPONYM-OF Color.

SYNONYM-OF

The realtionship between two keyphrases A and B is SYNONYM-OF if they both denote the same semantic field, for example Machine Learning SYNONYM-OF ML.