Natural Language to Code Generation on CoNaLa

Organized by edgarc - Current server time: Sept. 25, 2018, 5:36 p.m. UTC

Current

First phase
May 31, 2018, midnight UTC

End

Competition Ends
Never

CoNaLa

Welcome to the site of CoNaLa, the Code/Natural Language Challenge! This challenge was designed to test systems for generating programs from natural language. For example, if the input is "sort list x in reverse order", then the system would be required to output x.sort(reverse=True) in Python.

More information on the challenge, including data, etc. can be found here: https://conala-corpus.github.io/

 

Submitting for Evaluation

In your submission, please include a zipped .json file containing a list of hypothesis code strings. To submit, upload a .zip file with answer.txt containing the .json content, and please make sure you set your submission as public so it will appear on the list of submissions.

Evaluation consists of computing the BLEU score between the reference and hypothesis tokens. The tokens are treated as words.  The code will then be tokenized and compared with the reference. 

First phase

Start: May 31, 2018, midnight

Competition Ends

Never

You must be logged in to participate in competitions.

Sign In
# Username Score
1 anareshk 16.39
2 neubig 14.72