This is a competition based on the WiC (Word-in-Context) evaluation dataset (version 1.0).
Test data now available!
You can get all the information and data at https://pilehvar.github.io/wic. The evaluation period is currently open (see Evaluation tab for more details on how to submit your test results). For all questions regarding the evaluation you can write in the Google Group.
Google Group: https://groups.google.com/d/forum/WiC-Challenge
The current evaluation is part of a shared task co-located with the IJCAI workshop SemDeep-5. All participants are encouraged to write a short system description paper and to share their findings during the workshop (10-16 August 2019, exact date to be announced).
Note: If you downloaded the training data before 12 March 2019, please download it again as it has been updated.
12 March 2019: Test data available
12 March 2019: Evaluation start
12 April 2019: Evaluation end
26 April 2019: Short system description paper deadline
10-16 August 2019: IJCAI 2019, SemDeep workshop (exact date TBA)
Main reference paper:
Pilehvar, Mohammad Taher and Camacho-Collados, Jose (2019)
WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations
Proceedings of NAACL 2019. Minneapolis, USA.
Mohammad Taher Pilehvar, taher [dot] pilevar [at] gmail [dot] com
Jose Camacho-Collados, camachocolladosj [at] cardiff [dot] ac [dot] uk
In order to submit your test results, please create a file named "output.txt" with your answers, one per line ("T" if true or "F" if false), and then compress it into a .zip file. Then go to Participate (Submit / View results) -> Submit and upload your zipped system output file.
Each participant is allowed to submit a maximum of three systems/runs.
Start: Aug. 18, 2018, midnight
Start: March 12, 2019, midnight
Start: April 13, 2019, noon
You must be logged in to participate in competitions.Sign In