WiC-TSV: Word-in-Context Target Sense Verification
Previous
Test SemDeepCurrent
Test (no end)End
Competition EndsWelcome!
NEW! Final results of the SemDeep WiC-TSV challenge are available here. The CodaLab competition remains nevertheless open for further submissions.
This is a competition based on the WiC-TSV (Word-in-Context Target Sense Verification) evaluation benchmark. This competition extends the SuperGLUE WiC task but it is self-contained. The task models additional phenomena related to Word Sense Disambiguation, i.e., identifying the correct meaning of a word in context, in a retrieval setting. The main difference with WiC lies in the presence of relevant information such as hypernyms and definitions, which makes the task a direct proxy for downstream evaluation: in WiC-TSV a single word is presented with its context and relevant information, in contrast to two usages of the same word included in the original WiC dataset. This setting is arguably more realistic and resembles the usage of automatic tagging and retrieval in enterprise settings. For instance, an Indonesian company may want to retrieve all sentences referring to the Java island and not other unrelated senses.
WiC-TSV is used for a shared task at the IJCAI-20 SemDeep workshop. For questions about WiC-TSV, you can contact one of the organisers (information below).
NOTE: The dataset was updated on April 30. Test set contains both in-domain and out-of-domain subsets.
Task Details
Formally, WiC-TSV is framed as a binary classification task. Each instance in WiC-TSV consists of a target word w with a corresponding target sense s represented by either its definition (subtask 1) or its hypernym/s (subtask 2), and a context c containing the target word w. The task aims to determine whether the meaning of the word w used in the context c matches the target sense s. In the following table there are some examples from the dataset.
Data: Data is available in English - training, development and test data already available here. Data, code and more information also available in WiC-TSV github repo.
| Sentence | Definition | Hypernyms |
Tag |
|
| Smoking is permitted. |
The act of smoking tobacco or other substances | breathing, external respiration, respiration, ventilation |
True |
|
| All that work went down the sewer | Someone who sews | needleworker |
False |
|
Subtasks
WiC-TSV has three subtasks - participants can submit results in any of the subtasks:
Subtask 1: Definitions
In Subtask 1 systems make use of definitions for deciding whether the target word in context corresponds to the given definition or not.
Subtask 2: Hypernyms
In Subtask 2 systems make use of hypernymy information for deciding whether the target word in context is a hyponym of the given hypernym or not.
Subtask 3: Definitions + Hypernyms
In subtask 3 systems can make use of both sources of information, i.e., definitions and hypernyms.
Participation
Test data is provided and participants can submit their test results to the leaderboard. Participants can submit results in one, two or three of the subtasks. Submission is open indefinetely.
You can find information about participants to the first WiC-TSV challenge below:
- CTLR@WiC-TSV: Target Sense Verification using Marked Inputs and Pre-trained Models. Jose Moreno, Elvys Linhares Pontes and Gaël Dias. SemDeep-6 @ IJCAI 2020. [paper]
-
Word Sense Disambiguation with Transformer Models. Pierre-Yves Vandenbussche, Tony Scerri and Ron Daniel Jr. SemDeep-6 @ IJCAI 2020. [paper]
Reference paper
Please find below the reference paper for the WiC-TSV task with more details about the construction of the dataset and baselines (please cite it if you use WiC-TSV):
Anna Breit, Artem Revenko, Kiamehr Rezaee, Mohammad Taher Pilehvar and Jose Camacho-Collados (2021)
WiC-TSV: An Evaluation Benchmark for Target Sense Verification of Words in Context,
Proceedings of EACL 2021, to appear
Contact Info
|
Anna Breit |
Jose Camacho-Collados Cardiff University Mohammad Taher Pilehvar Iran University of Science and Technology |
Contact emails:
- anna.breit [at] semantic-web [dot] com
- camachocolladosj [at] cardiff [dot] ac [dot] uk
Sponsors:
Evaluation Criteria
Evaluation metrics: Accuracy and F-Measure (only accuracy in leaderboard).
In order to submit your test results, please create files named "definitions_output.txt", "hypernyms_output.txt" and/or "all_output.txt" (depending on the subtasks you are participating in) with your answers, one per line ("T" if true or "F" if false), and then compress it into a .zip file. These files should have the same number of lines as the test data. Then go to Participate (Submit / View results) -> Submit and upload your zipped system output file.
Development
Start: March 8, 2020, midnight
Test SemDeep
Start: Aug. 17, 2020, midnight
Test (no end)
Start: Sept. 22, 2020, midnight
Competition Ends
Never
You must be logged in to participate in competitions.
Sign In| # | Username | Score |
|---|---|---|
| 1 | gzervakis | 0.00 |