I submitted to test-dev2015 a couple hours ago, but my submission is stuck at "submitting." Is the evaluation server still running?
Posted by: daylen @ May 19, 2016, 7:08 p.m.CodaLab is having problems. I will follow up over email with details.
Posted by: vqateam @ May 19, 2016, 9:50 p.m.is there any update? my submissions are in "submitting" mode.
Posted by: omidb @ May 23, 2016, 11:33 p.m.I'm having the same issue. Can you guys fix this before the competition deadline?
Posted by: StevenReitsma @ May 24, 2016, 3:11 p.m.The codalab servers are having issues due to heavy submission traffic. We are in touch with Codalab and they are looking into it. We will keep you posted.
Posted by: vqateam @ May 25, 2016, 1:08 p.m.Few more servers have been added by Codalab people. Your submissions should get evaluated on time now.
Posted by: vqateam @ May 25, 2016, 9:34 p.m.In case your submissions are stuck in "submitting" mode for a long period of time, and you need results soon, you can email your results file (in the same format as required by evaluation server) to visualqa@gmail.com for manual evaluation. But you will still need to upload the files (especially if you want evaluation for test2015 or Challenge test2015 phases) to Codalab so that we can keep track of number of submissions and for sake of record. You would also want to upload to Codalab if you want your results to appear on leaderboard.
Posted by: vqateam @ May 27, 2016, 1:30 p.m.Hi,
Yestarday I uploaded a result for the Real Challenge test2015 (oe) and it's still in the submitting status. I will send the results to the email you provided to get the accuracy but my concern is if this results aren't evaluated before the challenge deadline, will it be entered in the challenge? I hope so.
Also, from the 5 results you allow to submit, I guess you are going to pick the one with higher accuracy, aren't you?
Thanks,
Posted by: xavigiro @ June 5, 2016, 6:13 p.m.I got back to you on email regarding your first question. Regarding second question, the entry with highest test-dev accuracy will be counted for challenge.
Posted by: vqateam @ June 6, 2016, 4:43 p.m.Submission stuck at "submitting" again (a few days), can you help check the evaluation server and fix the problem?
Posted by: dfyu @ July 5, 2016, 6:16 a.m.The issue was at the end of Codalab. It had to do with the fact that Azure modified one of their APIs. The issue has been resolved. Your new submissions should get processed on time now.
Posted by: vqateam @ July 7, 2016, 3:30 p.m.