1. Your system's output should contain the same number of lines as the input test data (make sure to verify this by running `wc -l`).
2. Each segment should read like normal Chinese.
3. You should rename your system's output file using the following convention: [AFFILIATION]_submission.txt, eg.: Didi_submission.txt. If you have multiple affiliations, you could concatenate each of them with an underscore '_'.
4. Create a README.txt which contains the details of your submission:
- Names of the team members
- Codalab username
- Affiliation(s)
- Method Description -- in one or 2 sentences.
5. Create a zip archive of the files in 3. and 4.
6. Send the submission (zip archive containing you system's output and the README.txt) to ajaynagesh@didiglobal.com (subject line should be "IWSLT 2020 JA --> ZH submission"). Please note if you are participating in ZH --> JA task, you need to send that submission separately (with the appropriate subject line).
[We will acknowledge the receipt of your submission as confirmation and intimate you if there are any issues with it]
Translation should be fully automatic. Participants should freeze translation systems before downloading the test data and should not manually tune or modify systems afterward.
Please note, the deadline for the submission is Apr 20, 2020 , 11.59 PM UTC-12h
(reposting here to ensure new entrants have access to this information)
Best,
Organizers
Thanks for submission details.
It shows that we should send the final zip file to ajaynagesh@didiglobal.com. We have three questions:
1. Is that means we should NOT submit the result in codaLab?
2. And is that means we cannot see the score of submission immediately?
3. How many chances we have? Can we resend email to overwrite previous submissions?
Thanks!
Posted by: casia @ March 28, 2020, 6:35 a.m.To answer your questions:
1. Is that means we should NOT submit the result in codaLab?
Ans: Please submit it directly via email. (we are facing some issues with Codalab setup, hence we thought it would be best to submit directly)
2. And is that means we cannot see the score of submission immediately?
Ans: Yes, you will not be able to see the submission score immediately. We will run eval on your submissions and intimate you in case there are any issues with your submission. We will make the final scores public at the end of the evaluation period.
3. How many chances we have? Can we resend email to overwrite previous submissions?
Ans: You have only 1 chance for a submission, which will be what you think is your best system output. Only in the rare case of any errors in your submission / very low bleu scores, we will contact you to resubmit.
Thanks for those clarification questions! I will add these FAQs and answers in the Submission Instructions page ("Participate" --> "Get Data and Submission Instructions")
Posted by: ajaynagesh @ March 28, 2020, 2:09 p.m.Dear Organizers:
I have some questions about post-processing:
1. Can we do post-processing based on development dataset, e.g. punctuations, numbers, full width and half width conversion?
2. If it is allowed, which kinds of punctuations and numbers should be submitted, full width or half width or keep the same with development dataset (since we have found its different between Chinese and Japanese)?
Thanks
To answer your questions:
We have released the evaluation script that we will be running on your submissions (in Codalab—> Learn the Details (tab) —> get_staring_kit —> Download Starting Kit). Beyond that we do not do any post-processing on your submissions. We cannot say anything about the reference files used in evaluation.
Posted by: ajaynagesh @ April 17, 2020, 12:32 a.m.