BioRelEx Benchmark 1.0

Organized by mahnerak - Current server time: Oct. 22, 2019, 5:44 a.m. UTC

Previous

Dev
July 1, 2019, midnight UTC

Current

Test
July 1, 2019, midnight UTC

End

Competition Ends
Dec. 31, 2050, midnight UTC

BioRelEx is a biological relation extraction dataset

Version 1.0 contains 2010 annotated sentences that describe binding interactions between various biological entities (proteins, chemicals, etc.). 

1405 sentences are for training, another 201 sentences are for validation. They are publicly available at https://github.com/YerevaNN/BioRelEx/releases

Another 404 sentences are for testing which are kept private for this Codalab competition. All sentences contain words "bind", "bound" or "binding".

For every sentence we provide:

  • Complete annotations of all biological entities that appear in the sentence
  • Entity types (32 types) and grounding information for most of the proteins and families (links to uniprot, interpro and other databases)
  • Coreference between entities in the same sentence (e.g. abbreviations and synonyms)
  • Binding interactions between the annotated entities
  • Binding interaction types: positive, negative (A does not bind B) and neutral (A may bind to B)

The goal of this challenge is to learn end-to-end relation extraction from raw sentences. 

Read more about the benchmark and the baselines in our paper at ACL Anthology.

Evaluation

The main evaluation metric is F-score of the extracted relations. We additionally report precision and recall. If the same entity is mentioned multiple times in the sentence (e.g. synonyms or abbreviations), we expect to the models to output relations between all possible versions. 

We do not have an explicit measure of coreference detection. It is implicitly captured in the relation extraction measures.

The evaluation ignores the labels of relations (positive, negative and neutral are all merged into a single class). It also ignores entity types and grounding information.

We also report precision, recall and F-score for entity mention detection. If the same entity is mentioned multiple times, we count all mentions of the entity (e.g. all synonyms and all mentions in the sentence).

The evaluation script is available at https://github.com/YerevaNN/BioRelEx

Read more about the evaluation and the baselines in our paper at ACL Anthology

No files have been added for this competition yet.

Dev

Start: July 1, 2019, midnight

Description: Development phase

Test

Start: July 1, 2019, midnight

Description: Test phase

Competition Ends

Dec. 31, 2050, midnight

You must be logged in to participate in competitions.

Sign In
# Username Score
1 mahnerak 50.15
2 mahnerak 39.10