Competition

Evaluation Criteria

1. Subtask 1: Sentence Classification
   We will report P/R/F1 for the positive and negative classes. The official score will be based on the F1 for the positive class.
2. Subtask 2: Sequence labeling
   We will report P/R/F1 for each evaluated class, as well as macro- and micro-averaged F1 for the evaluated classes. The official score will be based on the macro-averaged F1 of the evaluated classes. Evaluated classes include: Term, Alias-Term, Referential-Term, Definition, Referential-Definition, and Qualifier.
3. Subtask 3: Relation extraction
   We will report P/R/F1 for each evaluated relation, as well as macro- and micro-averaged F1 for the evaluated relations. The official score will be based on  the macro-averaged F1 of the evaluated relations. The evaluated relations include: Direct-defines, Indirect-defines, Refers-to, AKA, and Qualifies.

You can run these metrics locally by using the evaluation code available on the DEFT Github repo. The test set contains data from the same distribution of data as the train and dev sets. 

Running evaluation via Codalab

You may wish to run your evaluation via the Codalab framework to check your input formatting and to submit to the public leaderboard before the evaluation period begins. You must submit through the Codalab evaluation framework during the evaluation period in order for your submission to count towards the official competition ranking.

Please note the following phases and their corresponding evaluation data:

Practice Data: Trial data, found here

Training: Dev data, found here

All Evaluation phases: Test data, unlabeled data to be posted during the appropriate evaluation phase dates for each task.

To submit through Codalab, follow these steps: 

1. Navigate to Participate -> Submit/View Results
2. To test your results against the trial data (for formatting only), select 'Practice Data'.
   To test your results against dev data during the training period, select 'All tasks: training'.
   To submit your final results during the evaluation period, select the appropriate evaluation task. Note that the dates for evaluation of each task is staggered to prevent unfair advantages on any of the tasks.
3. Click 'Submit' to submit a .zip file containing your files to evaluate
   
   Your submission zip should include the .deft file for the task you want to evaluate and named using the following convention:
   task_[task number]_[name of source file].deft
   
   For example, if you are submitting files for task 1 to evaluate against the dev files, you would include the following files:
   task_1_t1_biology_0_0.deft
   task_1_t1_biology_0_202.deft
   ... 
   task_1_t7_government_2_303.deft
   
   If you are submitting multiple task files at once, simply include those files in the same submission.
   
   Be sure to double check to make sure you are using the correct data (ie., dev vs. training) as there are overlaps in the naming conventions across files in the separate sets.
   When submitting your zip file, you should zip only the files in your results folder. Do not submit a .zip file containing a folder.

Terms and Conditions

Data for this competition is comprised of annotations on excerpts from freely available textbooks at www.cnx.org. All data, including annotations, is provided under the CC-BY-NA 4.0 license.