Need about two or three days for submission once. It's too inefficient.
Posted by: dingzixiang @ May 15, 2021, 4:54 p.m.Hi dingzixiang,
Apologies for this. We’ve encountered a bug with Codalab which means that it can’t handle more than one machine without failing all submitted jobs. We’re hoping that this will be remedied shortly, in which case we can move to multiple evaluation machines.
Sorry for the inconvenience and thanks for your patience,
Rob
Hi,
I have a suggestion to make.
Judging from the series of failures on the manual queue, and also from my error logs,
I'm assuming most of us are encountering errors in the last dataset. (Maybe?)
So, in case parallelization of evaluation is not possible within few days or so,
why not start evaluating the last dataset first?
That way, the submissions that return errors for the last dataset will end up earlier than
waiting for the evaluation of first and second dataset to finish.
MLV, that's actually a brilliant idea, thanks so much. I've just made that change, hopefully the queue will start moving a lot quicker now. Additionally, I've also made sure the ingestion program prints out the shapes of each of the datasets and the dataset metadata, which should help people diagnose why the third dataset is causing them trouble.
Awesome! Thanks a ton :)
Posted by: MLV @ May 16, 2021, 9:01 a.m.