The final phase of the challenge will be run in 2 steps: Clicks then Sales.
As is it a bit cumbersome please take time to read this document carefully so that your best solutions are scored properly in the Final Leaderboards.
Generally speaking, the idea is that Codalab will display in the Final Leaderboards a single submission (i.e. the result of a single zip file) and select which one based on Clicks prediction performance. So if you wish to compete in the Sales leaderboard there are manuals steps to perform.
### Phase 1: Final Clicks Leaderboard (Aug 1st -> Aug 2nd) ###
Here is the timeline of events for the Clicks leaderboard:
1. Your top 2 submissions - as ranked on Clicks leaderboard of *Development phase* - will be automatically migrated to the Final Click Leaderboard (no action on your side is necessary). They will remain hidden for the moment.
2. You can *optionally* perform at most 2 additional, manual submissions to the Final phase between Aug 1st 00:00 UTC and Aug 2nd 00:00 UTC (do as usual if you were submitting new predictions). They will remain hidden for the moment.
3. Codalab will automatically select your best submission among the auto-migrated and manual submissions for the Final Click Leaderboard
4. The Final Clicks Leaderboard is revealed on Aug 2nd.
Please note submissions won't display their performance nor appear on the Final leaderboard until we activate the display of the leaderboard.
For Teams:
- Keep in mind Teams are ranked as a single participant: see this post https://competitions.codalab.org/forums/28181/5899/ and the Terms & Conditions
- Teams can submit a maximum of 2 manual submissions total (not 2 per team member - in case a team trespasses this limit we'll consider only the submissions of the best individual of the team, as ranked by the Clicks leaderboard in Dev and discard others)
### Phase 2: Final Sales Leaderboard (Aug 2nd -> Aug 3rd) ###
This step is _manual_ and **requires actions from participants** that wish to be scored properly on the Final Sales Leaderboard.
The idea is that for Codalab to properly handle your submission in both Clicks and Sales leaderboards you may need to craft a new submission combining your best Click predictions with your preferred Sales prediction.
At this point the Final Leaderboards are visible to all and the Clicks Leaderboard shall not change anymore. The Final Sales Leaderboard could still change before Aug 3rd depending on participants using their allowed quota of manual submissions.
Here is the procedure to craft your final, manual Sales submissions:
1. Select your best click submission as per the Final Clicks Leaderboards and extract the `y_hat_click.txt` file
2. Select up to 2 of your most promising Sales submissions and extract their `y_hat_sale.txt` file
3. Manually create up to 2 zip files containing both Clicks and Sales predictions (`y_hat_click.txt` and `y_hat_sale.txt`files) thus combining your best, final click submission with your most promising sales submissions
4. Upload at most 2 submissions to the Final Phase between Aug 2nd 00:00 UTC and Aug 3rd 00:00 UTC
5. Optionally, select your preferred submission to appear in Final Leaderboard before Aug 3rd 00:00 UTC (see https://competitions.codalab.org/forums/28181/5808/ for how to do that)
6. The Final Sales Leaderboard is revealed on Aug 3rd and the Competition is officially finished
Please make sure you use the `y_hat_click.txt` from your best submission in the final Clicks Leaderboard. Otherwise we can't guarantee that the Codalab platform will choose your best Sales prediction for the Final Sales Leaderboard.
For Teams the same rules apply: maximum 2 manual submissions per Team for the Final Sales Leaderboard.
### Remarks ###
- Please plan to connect to the competition website on the critical dates of Aug 2th and Aug 3th so as to be able to select your best submissions for the Final Leaderboards. We won't allow manual submissions after the Competition is finished.
- In total each participant will be able to submit at most 2 Clicks predictions and 2 Sales prediction in the Final phase. We will update the number of allowed submissions on Aug 2 and Aug 3 to allow for these submissions to be uploaded.
- We will check that participants don't submit Click predictions in the Sales upload period (Aug 2-3th) that weren't uploaded in the prior Click upload period (Aug 1-2th). Trespassing of this rule would be an undue advantage and may be considered as violation of the T&C.
- You can check manually the scoring of each of your submissions in the "Participate" > "Submit / View Results" menu and by clicking on the "+" sign on the right of your submission. To see all details, click "Download output from scoring step" and open the "scores.txt" file.
- If you have any questions please ask below in the thread or through our contact address (adkdd21challenge@googlegroups.com)
Hi! I'm trying to make submit on click leaderboard but unfortunately I get message "The maximum number of submissions has been reached.". But I did not any submissions during the final phase started. Could you please clarify what should I do? Thanks in advance.
Posted by: kss @ Aug. 1, 2021, 7:58 p.m.It was a configuration issue on our side. It should work now (starting Aug 2nd 5am UTC).
We will thus allow submissions for FINAL click phase until Aug 3rd 5am UTC to allow participants in any time zone to send their final submissions if they'd like to.
For FINAL Sales phase we'll also allow submissions from Aug 3rd 00am UTC until Aug 4th 5am UTC so that everybody can submit their best solution irrespective of their time zone.
Posted by: eustache @ Aug. 2, 2021, 5:31 a.m.Can confirm the issue still persists sadly. :(
Posted by: guruqu @ Aug. 2, 2021, 6:10 a.m.Might be the fact that all submission that was ever used in the leaderboard (in my case 25) are cloned to final leaderboard automatically which exceeds the 6 submission quota.
Posted by: guruqu @ Aug. 2, 2021, 6:17 a.m.Ah yes @guruqu, that must be it :-(
I've raised the configured limit just now, could you please try again ?
Posted by: eustache @ Aug. 2, 2021, 6:22 a.m.Worked.
Posted by: guruqu @ Aug. 2, 2021, 6:29 a.m.Click leaderboard still seems private, is it expected? :O
Posted by: guruqu @ Aug. 3, 2021, 5:34 a.m.It should be available now. The final checks, including respect of number of allowed submissions per team took a bit more time than expected !
Posted by: eustache @ Aug. 3, 2021, 6:17 a.m.I'm getting strange errors while submitting sale predictions even I'm just resubmitting my old archive with click and sale prediction, which was scored successfully some time ago.
AssertionError: invalid submission: expected one of y_hat_click.txt or y_hat_sale.txt files to be present
Can you please check this?
@or: your zip file is containing an "Archive" folder in which lie your prediction files. Your predictions files must be at the root of the zip file otherwise the scoring program is unable to find them.
Posted by: eustache @ Aug. 3, 2021, 9:19 a.m.@eustache: got it, thanks a lot for help.
Posted by: or @ Aug. 3, 2021, 9:37 a.m.