DeftEval 2020 (SemEval 2020 - Task 6)

Organized by sspala - Current server time: Sept. 19, 2019, 6:50 p.m. UTC


Practice Data
Aug. 15, 2019, midnight UTC


All tasks: Training
Sept. 4, 2019, midnight UTC


Subtask 1: Evaluation
Jan. 10, 2020, midnight UTC

DeftEval: Extracting term-definition pairs in free text


Welcome! Definition extraction has been a popular topic in NLP research for well more than a decade, but has been historically limited to well defined, structured, and narrow conditions. In reality, natural language is complicated, and complicated data requires both complex solutions and data that reflects that reality. The DEFT corpus expands on these cases to include term-definition pairs that cross sentence boundaries, lack explicit definitors, or definition-like verb phrases (e.g. is, means, is defined as, etc.), or appear in non-hypernym structures.


DeftEval is split into three subtasks

Subtask 1: Sentence Classification

Given a sentence, classify whether or not it contains a definition. This is the traditional definition extraction task.

Subtask 2: Sequence Labeling

Label each token with BIO tags according to the corpus' tag specification (see Data page).

Subtask 3: Relation Classification

Given the tag sequence labels, label the relations between each tag according to the corpus' relation specification (see Data page).

You may participate in any combination of the three subtasks, but note that the evaluation period for Subtask 3 will occur only after the end of the evaluation period for Subtask 2 in order to avoid any unfair release of test data.

Important Dates

  • Trial Data Release: 15 Aug 2019
  • Training Period: 04 Sept 2019
  • Subtask 1 Evaluation Period: 10 Jan 2020 - 20 Jan 2020
  • Subtask 2 Evaluation Period: 10 Jan 2020 - 20 Jan 2020
  • Subtask 3 Evaluation Period: 21 Jan 2020 - 31 Jan 2020

Task Organizers

  • Sasha Spala, Adobe Document Cloud, sspala at adobe dot com
  • Nicholas A Miller, Adobe Document Cloud
  • Franck Dernoncourt, Adobe Research,
  • Carl Dockhorn, Adobe Document Cloud

For questions and issues related to the task, please see the DeftEval-2020 forum. For questions and issues related to the data, please log issues on Github. To contact the organizers, please email the organizers at

Evaluation Criteria

More to come, stay tuned.

Terms and Conditions

Data for this competition is comprised of annotations on excerpts from freely available textbooks at All data, including annotations, is provided under the CC-BY-NA 4.0 license.

Practice Data

Start: Aug. 15, 2019, midnight

All tasks: Training

Start: Sept. 4, 2019, midnight

Subtask 1: Evaluation

Start: Jan. 10, 2020, midnight

Subtask 2: Evaluation

Start: Jan. 10, 2020, midnight

Subtask 3: Evaluation

Start: Jan. 21, 2020, midnight


Start: Feb. 1, 2020, midnight

Competition Ends


You must be logged in to participate in competitions.

Sign In