SemEval-2017 Task 1: Semantic Textual Similarity

Organized by ilopez077 - Current server time: March 24, 2018, 12:10 a.m. UTC

First phase

Aug. 1, 2016, midnight UTC


Competition Ends
Feb. 13, 2017, noon UTC
The competition is over! The provisional results are posted here
Check the task overview at

Explore the STS wiki


Organizers (alpha. order): Eneko Agirre, Daniel Cer, Mona Diab, Iñigo Lopez-Gazpio and Lucia Specia

Copyright 2017 - SemEval-2017 Task 1. All Right Reserved

Please read carefully the following instructions.There is no chance to replace submitted files and incorrect submissions won't be scored.
  1. Make sure you are registered to Participate in STS 2017
  2. Sign up in codalab
  3. You can use any username, but set the team name with care (in codalab profile settings) as this will be your official team name in the official results table and it will also be the name appearing in the title of the papers
  4. Download the test input pairs HERE
  5. Process the test input pairs with your system and send the results through codalab strictily following the submission naming convention (see below)
  6. You can submit a total of 3 runs. You may want to add a small description of the run in the textbox available above the submission button.
  7. The leaderboard will be hidden until the end of the competition
Submission naming convention
Each submission (run) consists of a .zip compressed file. This zip file can have any name, but it is recommended to use a meaningful name that describes the run. Avoid using strange characters.

 It is strongly recommended to participate in all subtracks, but you can just participate in the subtracks you wish. You will only obtain a primary track score if you participate in all subtracks.

The compresed .zip file must have one .txt file for each subtrack you wish to participate in, avoid adding any directory inside the .zip file.

The .txt files must contain one STS score per line and must have the following names:






Track5: STS.sys.track5.en-en.txt


Please ensure the naming is correct.

This is an example of a failed submission (run1 directory should not be included, use zip -j or -D if necessary to instruct zip not to create any directory):



Start: Aug. 1, 2016, midnight

Description: The site is non-stable and under construction.

Official evaluation

Start: Jan. 16, 2017, midnight

Description: Official evaluation

Submission correction

Start: Feb. 10, 2017, noon

Description: Phase to resend failed submissions (Double check submission instructions before submitting)

Competition Ends

Feb. 13, 2017, noon

You must be logged in to participate in competitions.

Sign In