BioRelEx Benchmark 1.0
Previous
DevCurrent
TestEnd
Competition EndsBioRelEx is a biological relation extraction dataset
Version 1.0 contains 2010 annotated sentences that describe binding interactions between various biological entities (proteins, chemicals, etc.).
1405 sentences are for training, another 201 sentences are for validation. They are publicly available at https://github.com/YerevaNN/BioRelEx/releases
Another 404 sentences are for testing which are kept private for this Codalab competition. All sentences contain words "bind", "bound" or "binding".
For every sentence we provide:
- Complete annotations of all biological entities that appear in the sentence
- Entity types (32 types) and grounding information for most of the proteins and families (links to uniprot, interpro and other databases)
- Coreference between entities in the same sentence (e.g. abbreviations and synonyms)
- Binding interactions between the annotated entities
- Binding interaction types: positive, negative (A does not bind B) and neutral (A may bind to B)
The goal of this challenge is to learn end-to-end relation extraction from raw sentences.
Read more about the benchmark and the baselines in our paper at ACL Anthology.
Evaluation
The main evaluation metric is F-score of the extracted relations. We additionally report precision and recall. If the same entity is mentioned multiple times in the sentence (e.g. synonyms or abbreviations), we expect to the models to output relations between all possible versions.
We do not have an explicit measure of coreference detection. It is implicitly captured in the relation extraction measures.
The evaluation ignores the labels of relations (positive, negative and neutral are all merged into a single class). It also ignores entity types and grounding information.
We also report precision, recall and F-score for entity mention detection. If the same entity is mentioned multiple times, we count all mentions of the entity (e.g. all synonyms and all mentions in the sentence).
The evaluation script is available at https://github.com/YerevaNN/BioRelEx
Read more about the evaluation and the baselines in our paper at ACL Anthology.
Sample submission is available at https://github.com/YerevaNN/biorelex-codalab-submission
Dev
Start: July 1, 2019, midnight
Description: Development phase
Test
Start: July 1, 2019, midnight
Description: Test phase
Competition Ends
Dec. 31, 2050, midnight
You must be logged in to participate in competitions.
Sign In| # | Username | Score |
|---|---|---|
| 1 | anonml1 | 67.09 |
| 2 | tuanlai245 | 62.18 |
| 3 | mahnerak | 50.15 |