SemEval 2019 Task 1: Cross-lingual Semantic Parsing with UCCA

Organized by borgr - Current server time: June 25, 2018, 3:43 p.m. UTC

Current

Practice
June 1, 2018, midnight UTC

Next

Evaluation
Jan. 10, 2019, midnight UTC

End

Competition Ends
Never

Welcome!

Semantic represetation is receiving growing attention in NLP in the past few years, and many proposals for semantic schemes have recently been put forth. Examples include Abstract Meaning Representation, Broad-coverage Semantic Dependencies, Universal Decompositional Semantics, Parallel Meaning Bank, and Universal Conceptual Cognitive Annotation. These advances in semantic representation, along with corresponding advances in semantic parsing, hold promise benefit essentially all text understanding tasks, and have already demonstrated applicability to summarization, paraphrase detection, and semantic evaluation (using UCCA; see below).

In addition to their potential applicative value, work on semantic parsing poses interesting algorithmic and modelling challenges, which are often different from those tackled in syntactic parsing, including reentrancy (e.g., for sharing arguments across predicates), and the modelling of the interface with lexical semantics. Semantic parsing into such schemes has been much advanced by recent SemEval workshops, including two tasks on Broad-coverage Semantic Dependency Parsing and two tasks on AMR parsing. We expect that a SemEval task on UCCA parsing to have a similar effect. Moreover, given the conceptual similarity between the different semantic representations, it is likely that work on UCCA parsing will directly contribute to the development of other semantic parsing technology. Furthermore, conversion scripts are available between UCCA and the SDP and AMR formats (note only the format is changed, we do not supply rules to convert one annotation to another, but you may create those).1 Teams that participated in past shared tasks on AMR and SDP, are encouraged to participate using similar systems and a conversion-based protocol.

UCCA is a cross-linguistically applicable semantic representation scheme, building on the established Basic Linguistic Theory typological framework. It has demonstrated applicability to multiple languages, including English, French and German (with pilot annotation projects on Czech, Russian and Hebrew), and stability under translation. It has proven useful for defining semantic evaluation measures for text-to-text generation tasks, including machine translation, text simplification and grammatical error correction.

UCCA supports rapid annotation by non-experts, assisted by an accessible annotation interface. The interface is powered by an open-source, flexible web-application for syntactic and semantic phrase-based annotation in general, and for UCCA annotation in particular.1

Task Definition

UCCA represents the semantics of linguistic utterances as directed acyclic graphs (DAGs), where terminal (childless) nodes correspond to the text tokens, and non-terminal nodes to semantic units that participate in some super-ordinate relation. Edges are labelled, indicating the role of a child in the relation the parent represents. Nodes and edges belong to one of several layers, each corresponding to a “module” of semantic distinctions.

UCCA’s foundational layer covers the predicate-argument structure evoked by predicates of all grammatical categories (verbal, nominal, adjectival and others), the inter-relations between them, and other major linguistic phenomena such as semantic heads and multi-word expressions. It is the only layer for which annotated corpora exist at the moment, and will thus be the target of this shared task. The layer’s basic notion is the Scene, describing a state, action, movement or some other relation that evolves in time. Each Scene contains one main relation (marked as either a Process or a State), as well as one or more Participants. For example, the sentence “After graduation, John moved to Paris” (see figure) contains two Scenes, whose main relations are “graduation” and “moved”. “John” is a Participant in both Scenes, while “Paris” only in the latter. Further categories account for inter-Scene relations and the internal structure of complex arguments and relations (e.g. coordination, multi-word expressions and modification).

UCCA distinguishes primary edges, corresponding to explicit relations, from remote edges (appear dashed in the figure) that allow for a unit to participate in several super-ordinate relations. Primary edges form a tree in each layer, whereas remote edges enable reentrancy, forming a DAG.

UCCA graphs may contain implicit units with no correspondent in the text. The figure shows the annotation for the sentence “A similar technique is almost impossible to apply to other crops, such as cotton, soybeans and rice.”. The sentence was used by to compare different semantic dependency schemes. It includes a single Scene, whose main relation is “apply”, a secondary relation “almost impossible”, as well as two complex arguments: “a similar technique” and the coordinated argument “such as cotton, soybeans, and rice.” In addition, the Scene includes an implicit argument, which represents the agent of the “apply” relation.

While parsing technology is well-established for syntactic parsing, UCCA has several distinct properties that distinguish it from syntactic representations, mostly UCCA’s tendency to abstract away from syntactic detail that does not affect argument structure. For instance, consider the following examples where the concept of a Scene has a different rationale from the syntactic concept of a clause. First, non-verbal predicates in UCCA are represented like verbal ones, such as when they appear in copula clauses or noun phrases. Indeed, in the figure, “graduation” and “moved” are considered separate Scenes, despite appearing in the same clause. Second, in the same example, “John” is marked as a (remote) Participant in the graduation Scene, despite not being explicitly mentioned. Third, consider the possessive construction in “John’s trip home”. While in UCCA “trip” evokes a Scene in which “John” is a Participant, a syntactic scheme would analyze this phrase similarly to “John’s shoes”.

The differences in the challenges posed by syntactic parsing and UCCA parsing, and more generally semantic parsing, motivate the development of targeted parsing technology to tackle it.

More information including the resources can be found in UCCA general resource page.

For more questions kindly look at other sections of the site. Questions that are left unanswered may be inquired in the dedicated group.


  1. https://github.com/huji-nlp/tupa/tree/master/scheme

  2. https://github.com/omriabnd/UCCA-App

Evaluation Criteria

Submission conditions.

Participants in the task will be evaluated in four settings:

  1. English in-domain setting, using the Wiki corpus.

  2. English out-of-domain setting, using the Wiki corpus as training and development data, and 20K Leagues as test data.

  3. German in-domain setting, using the 20K Leagues corpus.

  4. French setting with no training data, using the 20K Leagues as development and test data.

In order to allow both even ground comparison between systems and using hitherto untried resources, we will hold both an open and a closed track for submissions in the English and German settings. Closed track submissions will only be allowed to use the gold-standard UCCA annotation distributed for the task in the target language, and will be limited in their use of additional resources. Concretely, the additional data they will be allowed to use will only consist of that used by TUPA, which consists of automatic annotations provided by spaCy : 1 POS tags, syntactic dependency relations, and named entity types and spans. In addition, the closed track will allow the use of word embeddings provided by fastText 2 for all languages.

Systems in the open track, on the other hand, will be allowed to use any additional resource, such as UCCA annotation in other languages, dictionaries or datasets for other tasks, provided that they make sure not to use any additional gold standard annotation over the same text used in the UCCA corpora. 3 In both tracks, we will require that submitted systems will not be trained on the development data. Due to the absence of an established pilot study for French, we will only hold an open track for this setting.

The four settings and two tracks result in a total of 7 competitions, where a team may participate in anywhere between 1 and 7 of them. We will encourage submissions in each track to use their systems to produce results in all settings. In addition, we will encourage closed-track submissions to also submit to the open track.

For ease of submission in addition to the UCCA xml files conllu, sdp and amr formats will be allowed too, such submissions will be automatically converted to UCCA using this script.

To convert manually:
pip install semstr
python -m semstr.convert [filenames] -f [format] -o [out_dir]

Scoring.

In order to evaluate how similar an output UCCA structure is to a gold UCCA graph, we use DAG F1-score . Formally, over two UCCA annotations G1 and G2 that share their set of leaves (tokens) W and for a node v in G1 or G2 , define its yield (yield(v) subset or equal W) as its set of leaf descendants. Define a pair of edges ((v1,u1) in G1) and ((v2,u2) in G2) to be matching if (yield(u1) = yield(u2)) and they have the same label. Labeled precision and recall are defined by dividing the number of matching edges in G1 and G2  by |E1| and |E2| respectively. DAG F1-score is their harmonic mean.


  1. http://spacy.io

  2. http://fasttext.cc

  3. We are not aware of any such annotation, but include this restriction for completeness.

Terms and Conditions

This page enumerated the terms and conditions of the competition.

Practice

Start: June 1, 2018, midnight

Evaluation

Start: Jan. 10, 2019, midnight

Post-Evaluation

Start: Feb. 1, 2019, midnight

Competition Ends

Never

You must be logged in to participate in competitions.

Sign In