XL-WiC

Organized by tommasopasini - Current server time: Oct. 23, 2020, 10:27 p.m. UTC

Current

XL-WiC
Sept. 1, 2020, midnight UTC

End

Competition Ends
Never

Welcome!

This is a competition based on the WiC (Word-in-Context) evaluation task. This task consists of recognising whether a target word is used with the same meaning or not in two different contexts.

In XL-WiC there are test sets in many different languages. You can currently submit your test results to get your accuracy score in XL-WiC.

Please visit the official website for further information, including data and the official leaderboard.

The cross-Lingual setting uses the English WiC training and development datasets and tests the systems on either WordNet or Wiktionary datasets in different languages. In the monolingual setting, instead, the monolingual data for French, German and Italian provided with XL-WiC are used for training, development and test.

Accuracy is the official evaluation metric for all settings.

For a better understanding of the systems, the accuracy results in the monolingual setting will be further split into IV and OOV:

- IV indicates In-Vocabulary test sets. These contain only words that have been seen at training time.

- OOV indicates Out-Of-Vocabulary test sets. These contain only words that have not been seen at training time.

Terms and Conditions

The datasets in this competition are licensed under a Creative Commons Attribution-NonCommercial 4.0 License.

A submission is a zip file containing a txt file for each language and setting.

	
$ ls .
> submission.zip
$ unzip submission.zip
$ ls .
>.
 ├── submission.zip
 ├── bg_wn_xlingual.txt
 ├── da_wn_xlingual.txt
 ├── de_wikt_monolingual.txt
 ├── de_wikt_xlingual.txt
 ├── et_wn_xlingual.txt
 ├── fa_wn_xlingual.txt
 ├── fr_wikt_monolingual.txt
 ├── fr_wikt_xlingual.txt
 ├── hr_wn_xlingual.txt
 ├── it_wikt_monolingual.txt
 ├── it_wikt_xlingual.txt
 ├── ja_wn_xlingual.txt
 ├── ko_wn_xlingual.txt
 ├── nl_wn_xlingual.txt
 └── zh_wn_xlingual.txt
	

A submission can contain all the aforementioned files or only some of them. In this latter case only the submitted languages and setting will be evaluated.

Each file is a txt file and containing at the i-th line a 1 or a 0 indicating whether the corresponding item the i-th line in the test set is true or false, respectively. For example:

	
$ cat submission/bg_wn_xlingual.txt
1
0
0
0
1
...
	
	

XL-WiC

Start: Sept. 1, 2020, midnight

Description: The XL-WiC task consists of recognizing whether a target word is used with the same meaning or not in two different contexts. The 0-Shot setting uses the English WiC training and development data and tests the systems on either WordNet or Wiktionary datasets in different languages. In the monolingual setting, instead, the monolingual data for French, German and Italian provided with XL-WiC are used for training, development and test. IV indicates In-Vocabulary test sets. These contain only words that have been seen at training time. OOV indicates Out-Of-Vocabulary test sets. These contain only words that have not been seen at training time.

Competition Ends

Never

You must be logged in to participate in competitions.

Sign In