WiC competition

Organized by CamachoCollados - Current server time: May 19, 2019, 6:29 a.m. UTC


March 12, 2019, midnight UTC


April 13, 2019, noon UTC


Competition Ends

This is a competition based on the WiC (Word-in-Context) evaluation dataset (version 1.0).

Test data now available!

You can get all the information and data at https://pilehvar.github.io/wic. WiC was part of a shared task co-located with the IJCAI workshop SemDeep-5 (more details in the IJCAI WiC challenge tab). However, you can always submit your test output to the leaderboard (see Evaluation tab for more details on how to submit your test output). WiC is currently one of the datasets featured in the general language understanding framework SuperGLUE. For all questions regarding the evaluation you can write in the Google Group.

Google Group: https://groups.google.com/d/forum/WiC-Challenge

Note: If you downloaded the training data before 12 March 2019, please download it again as it has been updated.

Main reference paper:

Pilehvar, Mohammad Taher and Camacho-Collados, Jose (2019) 
WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations
Proceedings of NAACL 2019. Minneapolis, USA.

Contact Info:

Mohammad Taher Pilehvar, taher [dot] pilevar [at] gmail [dot] com

Jose Camacho-Collados, camachocolladosj [at] cardiff [dot] ac [dot] uk

Evaluation Criteria


In order to submit your test results, please create a file named "output.txt" with your answers, one per line ("T" if true or "F" if false), and then compress it into a .zip file (note that output.txt should have the same number of lines as the test data file). Then go to Participate (Submit / View results) -> Submit and upload your zipped system output file.

Each participant is allowed to submit a maximum of three systems/runs.

Important Dates:

12 March 2019: Test data available
12 March 2019: Evaluation start
12 April 2019: Evaluation end
16 April 2019: Results posted
3 May 2019 (extended)Short system description paper deadline
27 May 2019: Reviews available
10 June 2019: Camera-ready deadline
12 August 2019: SemDeep workshop, co-located with IJCAI 2019.

The final test results available of the WiC challenge are available here.


System description papers

System description papers of the IJCAI WiC challenge will be published in the ACL anthology

We are asking for 4-page system description papers (unlimited references), following the ACL guidelines. Word and LaTeX Templates can be downladed here. All papers should be submitted via EasyChair, using the following link. Some advice on how to write a system description paper can be found at the SemEval website

These papers will be peer-reviewed an additional page can exceptionally be given for the camera-ready version. More details about the WiC dataset (including more information about the dataset and the baselines) can be found in the reference paper. All authors are asked to check the paper and cite it in their system description papers:

  title={WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations},
  author={Pilehvar, Mohammad Taher and Camacho-Collados, Jose},
  booktitle={Proceedings of NAACL},
address = {Minneapolis, United States} }

All participants are encouraged to present their systems (either with a poster or with an oral presentation) at the IJCAI workshop SemDeep-5 (to be held before IJCAI in Macau, August 12).


Start: Aug. 18, 2018, midnight


Start: March 12, 2019, midnight


Start: April 13, 2019, noon

Competition Ends


You must be logged in to participate in competitions.

Sign In
# Username Score
1 dloureiro 68.07
2 CamachoCollados 50.00