WiC competition

Organized by CamachoCollados - Current server time: March 22, 2019, 6:10 p.m. UTC


Aug. 18, 2018, midnight UTC


March 12, 2019, midnight UTC


April 13, 2019, noon UTC

This is a competition based on the WiC (Word-in-Context) evaluation dataset (version 1.0).

Test data now available!

You can get all the information and data at https://pilehvar.github.io/wic. The evaluation period is currently open (see Evaluation tab for more details on how to submit your test results). For all questions regarding the evaluation you can write in the Google Group.

Google Group: https://groups.google.com/d/forum/WiC-Challenge

The current evaluation is part of a shared task co-located with the IJCAI workshop SemDeep-5. All participants are encouraged to write a short system description paper and to share their findings during the workshop (10-16 August 2019, exact date to be announced).

Note: If you downloaded the training data before 12 March 2019, please download it again as it has been updated.

Important Dates:

12 March 2019: Test data available
12 March 2019: Evaluation start
12 April 2019: Evaluation end
26 April 2019: Short system description paper deadline
10-16 August 2019: IJCAI 2019, SemDeep workshop (exact date TBA)


Main reference paper:

Pilehvar, Mohammad Taher and Camacho-Collados, Jose (2019) 
WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations
Proceedings of NAACL 2019. Minneapolis, USA.

Contact Info:

Mohammad Taher Pilehvar, taher [dot] pilevar [at] gmail [dot] com

Jose Camacho-Collados, camachocolladosj [at] cardiff [dot] ac [dot] uk

Evaluation Criteria


In order to submit your test results, please create a file named "output.txt" with your answers, one per line ("T" if true or "F" if false), and then compress it into a .zip file. Then go to Participate (Submit / View results) -> Submit and upload your zipped system output file.

Each participant is allowed to submit a maximum of three systems/runs.


Start: Aug. 18, 2018, midnight


Start: March 12, 2019, midnight


Start: April 13, 2019, noon

Competition Ends


You must be logged in to participate in competitions.

Sign In