Counts and measurements are an important part of scientific discourse. It is relatively easy to find measurements in text, but a bare measurement like "17 mg" is not informative. However, relatively little attention has been given to parsing and extracting these important semantic relations. This is challenging because the way scientists write can be ambiguous and inconsistent, and the location of this information relative to the measurement can vary greatly.
MeasEval is a new entity and semantic relation extraction task focused on finding counts and measurements, attributes of these quantities, and additional information including measured entities, properties, and measurement contexts.
(Updated 12 Nov 2020)
MeasEval is composed of five sub-tasks that cover span extraction, classification, and relation extraction, including cross-sentence relations. Note that all submissions will be evaluated against all five sub-tasks. Given a paragraph from a scientific text:
More detailed definitions can be found be reviewing our Annotation Guidelines.
Additional resources and data will be available on the MeasEval Github Repo
Register your team on the CodaLab Participate page.
Join our listserv at https://groups.google.com/forum/#!forum/measeval-semeval-2021
Corey Harper, Elsevier Labs and INDE lab at the University of Amsterdam
Jessica Cox, Elsevier Labs
Ron Daniel, Elsevier Labs
Paul Groth, INDE lab at the University of Amsterdam
Curt Kohler, Elsevier Labs
Antony Scerri, Elsevier Labs
(Updated 12 Nov 2020)
Evaluation will be based on a global F1 score averaged across all subtasks. For classification and relation extraction subtasks, this score will be a binary match score, while for span identification tasks, it will be based on a SQuAD style Overlap ("F1") score.
Although more granular scores will not be included in the leaderboard, the evaluation code can be executed locally to provide Exact Match scores for span identification tasks and P/R/F1 scores for all subtask components. For self-evaluation purposes prior to the test and evaluation period, the code can also be configured to provide scores averaged by docId (paragraph) or for each of nine separate score components of the five subtasks. These score components are: Quantity, Unit, Modifiers, MeasuredProperty, MeasuredEntity, Qualifier, HasQuantity, HasProperty, and Qualifies.
Data for this competition is in the form of annotations on CC-BY ScienceDirect Articles available from the Elsevier Labs OA-STM-Corpus. All data, including annotations, is provided under the CC-BY license.
The organizers make no warranties regarding the Dataset, including but not limited to being up-to- date, correct or complete.
By submitting results to this competition, you consent to the public release of your scores at SemEval2021 and in related publications.
Start: Oct. 1, 2020, midnight
Start: Jan. 10, 2021, midnight
Start: Feb. 1, 2021, midnight
You must be logged in to participate in competitions.Sign In