Natural Language to Code Generation on CoNaLa
Current
First phaseEnd
Competition EndsCoNaLa
Welcome to the site of CoNaLa, the Code/Natural Language Challenge! This challenge was designed to test systems for generating programs from natural language. For example, if the input is "sort list x in reverse order", then the system would be required to output x.sort(reverse=True) in Python.
More information on the challenge, including data, etc. can be found here: https://conala-corpus.github.io/.
Submitting for Evaluation
In your submission, please include a zipped .json file containing a list of hypothesis code strings. To submit, upload a .zip file with answer.txt containing the .json content, and please make sure you set your submission as public so it will appear on the list of submissions.
Evaluation consists of computing the BLEU score between the reference and hypothesis tokens. The tokens are treated as words. The code will then be tokenized and compared with the reference.
First phase
Start: May 31, 2018, midnight
Competition Ends
Never
You must be logged in to participate in competitions.
Sign In| # | Username | Score |
|---|---|---|
| 1 | neubig | 14.72 |