Competition

Evaluation Metrics

For this task, we adopt three metrics for the Product1M evaluation, i.e., Precision(Prec@N), mean Average Precision(mAP@N) and mean Average Recall (mAR@N).

• Precision(%) for information retrieval, used in the Image Retrieval
• Mean Average Precision(%) for information retrieval, used in the Google Landmark
• Mean Average Recall(%) takes the category distribution into account, i.e., the inclusion of instance ratio is informative for evaluating both the correctness and diversity of a retrieval algorithm and guarantees that some trivial results are not overestimated. mAR is computed as follow

More detail about mAR can be seen in Google doc.

Submission Format

The results should be written into a .txt file named retrieval_results.txt and then archived into a ZIP file (retrieval_results.zip). The example text file is available in retrieval_results.txt.

Specifically, each line in the text file contains the query id followed by a ranked retrieval id list, which is limited to 100 ids since we only evaluate at most `N=100` for all three metric.
Examples in the text file: `id_q,id_0,id_1,id_2,...,id_100`

After uploading your results, please wait for 40 to 50 minutes and refrash your page to see the scores.

Terms and Conditions

General Rules

• Each entry is required to be associated to a team and its affiliation.
• Using multiple accounts to increase the number of submissions is strictly prohibited.
• Results should follow the correct format and must be uploaded to the evaluation server through the CodaLab competition site. Detailed information about how results will be evaluated is represented on the evaluation page.
• The best entry of each team will be public in the leaderboard at all time.
• The organizer reserves the absolute right to disqualify entries which is incomplete or illegible, late entries or entries that violate the rules.

Datasets

The datasets are released for academic research only and it is free to researchers from educational or research institutions for non-commercial purposes. When downloading the dataset you agree not to reproduce, duplicate, copy, sell, trade, resell or exploit for any commercial purposes, any portion of the images and any portion of derived data.

LICENSE

PRODUCT-1M DATASET LICENSE

COMMITMENT LETTER

Alibaba Group is the only owner of all intellectual property rights (including copyright) of the PRODUCT-1M DATASET (“Dataset” or “Data”). Alibaba Group reserves the right to terminate Licensee’s access to the Dataset at any time. This PRODUCT-1M DATASET LICENSE COMMITMENT LETTER (“Letter”) explains the rules before and after I/we (“Licensee”) download and use the PRODUCT-1M DATASET. By downloading or using the Dataset, as a Licensee I/we understand, acknowledge, and hereby agree to all the terms in this Letter.

I. APPLICATION REQUIREMENTS

The permission for application is only open to researchers or faculties of universities or research institutes who have successfully signed up for the Weakly-supervised Product Retrieval Competition (“Competition”). Alibaba Group reserves the right to distribute the license.

II. NO COMMERCIAL USE

Licenses free of charge are limited to non-commercial research use only. Licensee is only granted a limited, non-exclusive, non-assignable, and non-transferable license to use the Dataset, which cannot apply for any commercial use.

III. NO DISTRIBUTION

This license is not a sale of any or all of the owner’s rights. Licensee is not allowed to sublicense or distribute the Dataset in whole or in part to any third party. Licensee guarantees that the Dataset may only be used by himself, and Licensee cannot rent, lease, lend, sub-license, or transfer the Dataset or any rights under this Letter to anyone or any third party else.

IV. RESTRICTED USE IN RESEARCH

Licensee guarantees that the Dataset can only be used for essential purposes related to the Competition during the competition, such as analysis, experimentation, and display of competition results. Besides, it is prohibited for any business entity to obtain or use the Dataset.

V.LEGAL LIABILITY

Licensee shall indemnify, defend and hold harmless Alibaba Group, their directors, employees and representatives, from and against any and all claims arising out of Licensee’s use of Dataset. And Licensee is fully aware that Licensee shall make compensation for all the “Losses” suffered by Alibaba Group arising out of Licensee’s breach of any term of this Letter. The term “Losses” shall include, but not limited to: Any refunds, liquidated damages or compensation that Alibaba Group paid to third parties; Penalties; Legal fees and other fees and expenses incurred by Alibaba Group to eliminate, mitigate and/or otherwise manage the impact on Alibaba Group arising out of or related to Licensee’s act or failure to act.

Concate Us

For more information, please concate us at zhanxlin@mail2.sysu.edu.cn or dx.icandoit@gmail.com.