SemEval-2017 Task 3 Subtask A
Previous
TestingCurrent
DevelopmentEnd
Competition EndsWelcome!
Community Question Answering (CQA) forums are gaining popularity online. They are seldom moderated, rather open, and thus they have few restrictions, if any, on who can post and who can answer a question. On the positive side, this means that one can freely ask any question and expect some good, honest answers. On the negative side, it takes effort to go through all possible answers and to make sense of them. For example, it is not unusual for a question to have hundreds of answers, which makes it very time consuming to the user to inspect and to winnow. The challenge we propose may help automate the process of finding good answers to new questions in a community-created discussion forum (e.g., by retrieving similar questions in the forum and identifying the posts in the answer threads of those questions that answer the question well).
We build on the success of the previous editions of our SemEval tasks on CQA, SemEval-2015 Task 3 and SemEval-2016 Task 3, and present an extended edition for SemEval-2017, which incorporates several novel facets.
This CodaLab competition is for Subtask A of SemEval Task 3: the Question-Comment Similarity Subtask (English)
This task is the same as our main CQA task in 2016.
Given:
- a new question (aka the original question),
- its first 10 comments in the question thread,,
rerank these 10 comments according to their relevance with respect to the question.
We want the "Good" comments to be ranked above the "PotentiallyUseful" or "Bad" comments; the latter two will not be distinguished and will be considered "Bad" in terms of evaluation. The gold labels for this subtask are contained in the RELC_RELEVANCE2RELQ field of the XML files. See the datasets README file for a detailed description of the XML file format. Although we target the first 10 comments in terms of their time of posting (rather than in terms of their relevance), the output of the system will be a ranked list of comments (in terms of probability of relevance). Therefore, this is a ranking task, not a classification task.
More information on the task and all the subtasks can be found on the SemEval Task website.
Evaluation Criteria
On the Leaderboard three scores will be provided: MAP, Average Recall, and MRR, but the official evaluation measure towards which all systems will be evaluated and ranked will be mean average precision (MAP) using the 10 ranked comments.
Note: the dataset is formatted for training in this subtask. The format required for the output of your systems will be detailed in the scorer and format-checker README files. These can be found here.
The name of the development file you submit needs to be SemEval2017-Task3-CQA-QL-dev-subtaskA.xml.subtaskA.pred, and it needs to be zipped.
The name of the test file you submit needs to be SemEval2017-task3-English-test-subtaskA.xml.subtaskA.pred, and it needs to be zipped.
Terms and Conditions
By participating in this competition and submitting results in CodaLab you agree to the public release of your results in the proceedings of SemEval 2017. Furthermore, you accept that the choice of evaluation metric is made by the task organizers, and that they have the right to decide the winner of the competition, and to disqualify teams if they do not follow the rules of the competition.
Development
Start: Aug. 1, 2016, midnight
Description: The development phase
Testing
Start: Jan. 9, 2017, midnight
Description: The testing phase
Competition Ends
Jan. 31, 2017, noon
You must be logged in to participate in competitions.
Sign In| # | Username | Score |
|---|---|---|
| 1 | kath | 0.717 |
| 2 | Mr_Woo | 0.714 |
| 3 | TitasNandi | 0.676 |