WiC-TSV: Word-in-Context Target Sense Verification

Organized by CamachoCollados - Current server time: April 3, 2020, 5:08 a.m. UTC

Current

Development
March 8, 2020, midnight UTC

Next

Test SemDeep
April 10, 2020, midnight UTC

End

Competition Ends
Never

Welcome!

This is a competition based on the WiC-TSV (Word-in-Context Target Sense Verification) evaluation benchmark. This competition extends the SuperGLUE WiC task but it is self-contained. The task models additional phenomena related to Word Sense Disambiguation, i.e., identifying the correct meaning of a word in context.  The main difference with WiC lies in the presence of relevant information such as hypernyms and definitions, which makes the task a direct proxy for downstream evaluation: in WiC-TSV a single word is presented with its context and relevant information, in contrast to two usages of the same word included in the original WiC dataset. This setting is arguably more realistic and resembles the usage of automatic tagging in enterprise settings. For instance, an Indonesian company may want to retrieve all sentences referring to the Java island and not other unrelated senses.

WiC-TSV is used for a shared task at the IJCAI-20 SemDeep workshop. For questions about WiC-TSV, you can contact one of the organisers (information below).

Task Details

Formally, WiC is framed as a binary classification task. Each instance in WiC-TSV consists of a target word w with a corresponding target sense s represented by either its definition (subtask 1) or its hypernym/s (subtask 2), and a context c containing the target word w. The task aims to determine whether the meaning of the word w used in the context c matches the target sense s. In the following table there are some examples from the dataset. Data is available in English - training and development data already available here.

Sentence  Definition Hypernyms

Tag

Smoking is permitted.   
 The act of smoking tobacco or other substances breathing, external respiration, respiration, ventilation 

True

All that work went down the sewer  Someone who sews  needleworker

False

 

 

 

Subtasks

 WiC-TSV has three subtasks - participants can submit results in any of the subtasks:

Subtask 1: Definitions

In Subtask 1 systems make use of definitions for deciding whether the target word in context corresponds to the given definition or not.

Subtask 2: Hypernyms

In Subtask 2 systems make use of hypernymy information for deciding whether the target word in context is a hyponym of the given hypernym or not.

Subtask 3: Definitions + Hypernyms

In subtask 3 systems can make use of both sources of information, i.e., definitions and hypernyms.

Participation

During the test phase, test data will be provided and participants can submit their results to the leaderboard. Participants can submit results in one, two or three of the subtasks. A maximum of two outputs per subtask is allowed. Attendance to the SemDeep workshop is encouraged to share the results but not mandatory. All participants, regardless of whether they attend the workshop or not, will be asked to write a small system description paper (up to four pages).

Contact Info

Anna Breit
Artem Revenko

Semantic Web Company

Jose Camacho-Collados
Cardiff University

Mohammad Taher Pilehvar
Iran University of Science and Technology

Contact emails:

anna.breit [at] semantic-web [dot] com
- camachocolladosj [at] cardiff [dot] ac [dot] uk

Sponsors:

 

 

Evaluation Criteria

Accuracy.

In order to submit your development results, please create files named "definitions_output.txt", "hypernyms_output.txt" and/or "all_output.txt" (depending on the subtasks you are participating in) with your answers, one per line ("T" if true or "F" if false), and then compress it into a .zip file. These files should have the same number of lines as the test data. Then go to Participate (Submit / View results) -> Submit and upload your zipped system output file.

Important Dates:

10 March 2020: Training data release
10 April 2020: Test data release. Evaluation start
1 May 2020: Evaluation end
4 May 2020: Results available
15 May 2020: System description paper deadline
1 June 2020: Author notifications with reviews
12 Jun 2020: Camera-ready submission deadline
11-13 Jul 2020 (to be specified): SemDeep Workshop, co-located with IJCAI

Development

Start: March 8, 2020, midnight

Test SemDeep

Start: April 10, 2020, midnight

Test (no end)

Start: May 2, 2020, midnight

Competition Ends

Never

You must be logged in to participate in competitions.

Sign In