Competition

BEA 2019 Shared Task: Grammatical Error Correction

Building Educational Applications 2019 Shared Task:
Grammatical Error Correction

RESTRICTED TRACK

Welcome to the BEA 2019 Shared Task on Grammatical Error Correction: Restricted Track. Please, check the shared task website if you are looking for a different track.

On this website, you will be able to evaluate your system output on the development set and make a submission for the test set when the test phase opens. After the shared task is finished, this website will remain open so that anyone can evaluate their system output and compare their performance with other teams.

• Shared task website: https://www.cl.cam.ac.uk/research/nl/bea2019st/
• Google Group: bea2019st
• Input data for submissions: development | test

Phases

The competition is composed of three phases: development, test and open.

During the development phase, participants will be able to evaluate their systems on the W&I+LOCNESS development set to help them develop their systems. This will also give participants the opportunity to become more familiar with the Codalab submission procedure in advance of the test phase. The development phase will remain open indefinitely and participants can make as many submissions as they like (up to 999). For each successful submission, participants will be able to see their overall and detailed results.

During the test phase, participants will be able to make up to 2 submissions per team on the official W&I+LOCNESS test set between 25th March 2019 00:00 UTC and 29th March 2019 23:59 UTC. This is the official evaluation phase of the shared task, so participants will not be able to see their results until after the end of the phase and the official results are announced. They can at least make sure that their submission status is "Finished" (i.e. successful) however.

During the open phase, participants will continue to be allowed to make submissions on the official W&I+LOCNESS test set, however these submissions will not be included in the official BEA2019 shared task results. The main aim of the open phase is to encourage future work into GEC beyond the shared task. The open phase will remain open indefinitely and participants can make as many submissions as they like (up to 999). In addition to being able to see their own overall and detailed results, participants will also be able to add their scores to the leaderboard if they want to make them publicly available.

Evaluation

Systems will be evaluated in terms of span-based correction F_0.5 using the ERRANT scorer. For further details, please refer to the shared task website.

Registration

Although we do not require participants to register for the shared task, Codalab enforces registration in order to participate. To do this, go to the "Participate" tab, accept the terms and conditions and click on "Register". Registration is automatic so you do not need to wait for approval (despite what Codalab says).

Team names

Before making a submission, we strongly recommend that you set up a name for your team. The easiest way to do this is to go to your Codalab profile settings and fill in the "Team name" field under the "Competition settings" section. Note, however, that this is a global team name that will be used for all the competitions you participate in.

If you prefer setting up a specific team name for this competition, you must first register as described above and then create a new team via the "Team" tab that will appear. This is a more advanced option that will let you use a logo and add members to your team, etc. If you make use of this option, please make sure you have properly set up your team name before making a submission, as it is not possible to assign a competition-specific team name to a submission after it has been made.

Full instructions on how to set up team names can be found here.

Making a submission

First of all, make sure that you have used the right input data for the phase you are submitting to. Links to the development and test data are provided at the top of this page. These files contain the tokenised input sentences that your system should correct. The output file produced by your system should contain corrected versions of the same sentences, also tokenised and one per line. For example:

Input files were tokenised using spaCy v1.9.0 and the en_core_web_sm-1.2.0 model so we strongly recommend that you use the same settings for producing your output files.

In order to make a submission to this competition, you must first zip your output file. The name of the files is not important, but you should make sure that the zipped file only contains a single file. It should also not contain any directories or subdirectories.

To upload your submission zip file, go the the "Participate" tab and click on the phase you want to submit to. If the phase is open, you will see a form with a "Submit" button to upload your submission. You should be able to see the status of your submission at different stages. If it is successful, the "Status" column will show "Finished", otherwise, it will show "Failed" and not count towards your total number of submissions.

If you are submitting a file during the development or open phase, you will be able to view your main F_0.5 score ("Score" column), overall results ("View scoring output log"), detailed results ("View detailed results") and any error messages ("View scoring error log"). During the open phase, you can add your results to the public leaderboard by clicking on the "Submit to Leaderboard" button.

If you encounter any problems during your submission or something is not working as expected, please contact the shared task organisers by email or post a question in our forum.