Abstract Meaning Representation (AMR) is a compact, readable, whole-sentence semantic annotation. Annotation components include entity identification and typing, PropBank semantic roles, individual entities playing multiple roles, entity grounding via wikification, as well as treatments of modality, negation, etc.
Here is an example AMR for the sentence “The London emergency services said that altogether 11 people had been sent to hospital for treatment due to minor wounds.”
(s / say-01 :ARG0 (s2 / service :mod (e / emergency) :location (c / city :wiki ‘‘London’’ :name (n / name :op1 ‘‘London’’))) :ARG1 (s3 / send-01 :ARG1 (p / person :quant 11) :ARG2 (h / hospital) :mod (a / altogether) :purpose (t / treat-03 :ARG1 p :ARG2 (w / wound-01 :ARG1 p :mod (m / minor)))))
Note the inclusion of PropBank semantic frames (‘say-01’, ‘send-01’, ‘treat-03’, ‘wound-01’), grounding via wikification (‘London’), and multiple roles played by an entity (e.g. ‘11 people’ are the ARG1 of send-01, the ARG1 of treat-03, and the ARG1 of wound-01).
In 2016 SemEval held its first AMR parsing challenge and received strong submissions from 11 diverse teams. In 2017 we have extended the challenge to both parsing of biomedical data and generation. This subtask is concerned with the former:
Subtask 1: Parsing Biomedical Data
As in 2016, participants will be provided with parallel English-AMR training data. They will have to parse new English data and return the obtained AMRs. The genre of the data is quite different from that in 2016. It focuses on scientific articles regarding cancer pathway discovery.
Here is an example parse of the sentence "Among tested agents, the B-Raf inhibitor dabrafenib was found to induce a strong V600E-dependent shift in cell viability."
(f / find-01 :ARG1 (i2 / induce-01 :ARG0 (s / small-molecule :name (n3 / name :op1 "dabrafenib") :ARG0-of (i3 / inhibit-01 :ARG1 (e2 / enzyme :name (n2 / name :op1 "B-Raf"))) :ARG1-of (i4 / include-01 :ARG2 (a / agent :ARG1-of (t2 / test-01)))) :ARG2 (s2 / shift-01 :ARG1 (v / viability :mod (c / cell)) :ARG0-of (d / depend-01 :ARG1 (m / mutate-01 :value "V600E")) :mod (s3 / strong))))
Participants may use any resources at their disposal (but may not hand-annotate the blind data or hire other human beings to hand-annotate the blind data). The SemEval trophy goes to the system with the highest Smatch score.
More example Bio data with AMRs can be found here
Participation is a two-phase process:
Participation in each phase is more or less the same:
The primary trophy-determining metric for this subtask will be the automated metric Smatch, commit fda1c9ea564a142d2dd0a6455627e69348662c9b, the version from 2016-11-14, located at https://github.com/snowblink14/smatch. This is subject to change.
Other automated metrics, such as other versions of Smatch, and sub-evaluations such as Smatch calculated on specific parts of the corpus will also be displayed in the online submission system and may be part of SemEval workshop proceedings. These metrics are not official and not trophy-determining.
We welcome the proposal of other human and automated metrics for this task, since it is not at all clear that the above proposed methods are in fact the best way to evaluate systems. That being said, unless otherwise indicated by the task organizer, the trophy-determining metric is that listed above.
By submitting to the 'Evaluation' phase of this track you agree to the public release of your submissions' scores at the SemEval 2017 workshop and in the associated publicly available proceedings, at the task organizer's discretion. Scores may include, but are not limited to, automatic and manual quantitative judgements, qualitative judgements, and other metrics as the task organizer sees fit. You accept that the ultimate decision of metric choice and score value is that of the task organizer. You further agree that your system will be named according to the team name provided at the time of submission or to a suitable shorthand, as determined by the task organizer. You agree that the task organizer is under no obligation to release scores and that scores may be withheld if it is the task organizer's judgement that the submission was incomplete, deceptive, or violated the letter or spirit of the competition's rules. Inclusion or exclusion of a submission's scores is not an endorsement or unendorsement of a team or individual's submission, system, or science. You further acknowledge that all trophy-making decisions are made at the sole discretion of the task organizer and that the organizer may present zero or more trophies. The definition of what constitutes a trophy is up to the task organizer.
Start: Aug. 1, 2016, midnight
Description: Parse News/Forum data from 2016 SemEval Task 8 (LDC2016E33) and/or the Bio AMR Corpus version 0.8 test subcorpus. News/Forum data is included with the Newswire/Discussion Forum data pack released by LDC and Bio data is publicly available. Follow the instructions provided in the 'Get Data' tab to obtain it. See 'Evaluation' under 'Learn the Details' for information on how to submit.
Start: Jan. 9, 2017, midnight
Description: Parse the SemEval 2017 Task 9 Bio AMR Evaluation corpus. This data will be released when the evaluation period begins. See 'Evaluation' under 'Learn the Details' for information on how to submit.
Jan. 20, 2017, midnight
You must be logged in to participate in competitions.Sign In