Evaluation Metrics

The retrieval results are evaluated by recall on the test splits of the dataset. For each of the three fashion categories (dresses, tops, shirts), Recall@10 and Recall@50 will be computed on all test queries. The overall performance is computed by taking the average of Recall@10 and Recall@50 on the three fashion categories.

Submission Format

Please save your results in three files: "dress.predict.json", "toptee.predict.json", "shirt.predict.json" for both the development submission and the test submission. Please zip all three files into one "zip" file and submit it to the leaderboard of the respective phase. For each query, you will only need to keep the top 50 image IDs of the retrieval ranking list, since we only need to compute recall@10 and recall@50.

The submission should use the following format:

[{

"candidate": image_ID(str),

"ranking": image_ID(str),image_ID(str),image_ID(str),....

},

{

"candidate": image_ID(str),

"ranking": image_ID(str),image_ID(str),image_ID(str),....

},

... ...

]

Additional Information

• We provide the evaluation script for your reference.
• This is a sample submission. It contains "val.zip" for the dev phase and "test.zip" for the final test phase. Please unzip the sample submission and submit "val.zip" to dev leaderboard to check if your account works properly.

Terms and Rules

General Rules

• We will offer cash awards to top contenders of the leaderboard. 
• Participants must submit a technical report and code repo to describe their methods by May. 24, 2020. The technical reports and code repo will be linked on the workshop website. 
• Participants can leverage external data or pretrained networks. Any and all external data used for training must be specified in the submitted technical report. However, external data or side information (attribute tags, Amazon product metadata, etc.) can not be used during the testing phase.
• Results should follow the correct format and must be uploaded to the evaluation server through the CodaLab competition site.
• Each entry needs to be associated to a team and its affiliation.
• The organizers reserve the right to disqualify entries which is incomplete or illegible, late entries or entries that violate the rules.
• The best entry of each team will be public in the leaderboard at all time. 

Dataset Access and Usage

• The datasets are released for academic research only and it is free to researchers from educational or research institutions for non-commercial purposes. When downloading the dataset you agree not to reproduce, duplicate, copy, sell, trade, resell or exploit for any commercial purposes, any portion of the images and any portion of derived data.
• To obtain the attribute labels associated with Fashion IQ dataset, please send an email to yupeng.gao@ibm.com 
• Please cite the following paper if are using Fashion IQ dataset: 

The Fashion IQ Dataset: Retrieving Images by Combining Side Information and Relative Natural Language Feedback. Xiaoxiao Guo*, Hui Wu*, Yupeng Gao, Steve Rennie and Rogerio Feris. Arxiv. 2019