Competition

The cross-Lingual setting uses the English WiC training and development datasets and tests the systems on either WordNet or Wiktionary datasets in different languages. In the monolingual setting, instead, the monolingual data for French, German and Italian provided with XL-WiC are used for training, development and test.

Accuracy is the official evaluation metric for all settings.

For a better understanding of the systems, the accuracy results in the monolingual setting will be further split into IV and OOV:

- IV indicates In-Vocabulary test sets. These contain only words that have been seen at training time.

- OOV indicates Out-Of-Vocabulary test sets. These contain only words that have not been seen at training time.

Terms and Conditions

The datasets in this competition are licensed under a Creative Commons Attribution-NonCommercial 4.0 License.

A submission is a zip file containing a txt file for each language and setting.

	
$ ls .
> submission.zip
$ unzip submission.zip
$ ls .
>.
 ├── submission.zip
 ├── bg_wn_xlingual.txt
 ├── da_wn_xlingual.txt
 ├── de_wikt_monolingual.txt
 ├── de_wikt_xlingual.txt
 ├── et_wn_xlingual.txt
 ├── fa_wn_xlingual.txt
 ├── fr_wikt_monolingual.txt
 ├── fr_wikt_xlingual.txt
 ├── hr_wn_xlingual.txt
 ├── it_wikt_monolingual.txt
 ├── it_wikt_xlingual.txt
 ├── ja_wn_xlingual.txt
 ├── ko_wn_xlingual.txt
 ├── nl_wn_xlingual.txt
 └── zh_wn_xlingual.txt
	

A submission can contain all the aforementioned files or only some of them. In this latter case only the submitted languages and setting will be evaluated.

Each file is a txt file and containing at the i-th line a 1 or a 0 indicating whether the corresponding item the i-th line in the test set is true or false, respectively. For example:

	
$ cat submission/bg_wn_xlingual.txt
1
0
0
0
1
...