Hi
Do you think that in the current situation extension of the evaluation period will be reasonable?
We have just uploaded the file with results, but none of the patent numbers seems valid. Which of the files provided by you contains the numbers of patents in the evaluation set?
Cheers,
Bozena
Hi
Your submissions runs at my PC, but something is odd (78 ids are missing ). I runs with the development dataset: Score is Task A, F1-score: 0.028, Task B, F1-score: 0.027. Probably you mixed up the ids of dev and test (predictions for test with ids from dev?). Can you check that?
Extension seems likely, the codalab is already set for it. I will not rule out, although now it is a little early.
Cheers,
Fernando
Thank you for quick reaction.
"mixed prediction with test with ids for dev " -which test do you mean? I used dev as text set, and found in the rest of the dataset (descriptions, titles, abstracts) patents with the numbers listed in dev. I made predictions based on this.
How do you calculate F1 ? In the training phase I used f1_score from sklearn.metrics, it this one appropriate here?
Cheers
Bozena
Sorry, I did change the participate page and put only there the link to the test.csv but I did not change the data description page. Here the link to the test.csv file:
https://filer.cloudlab.zhaw.ch/index.php/s/zRtzDyaPHdnmSYP
We are in the Test phase now, and you uploaded to the test set (which you can get now...). Thus, it expects the Ids from test.csv. The dev.csv was for the development phase.
Sorry for the mix up!
I will upload soon the evaluate.py script (It needs some editing because of the zero-shot task).
Cheers,
Fernando
Yes, I can see now; our submission shall be for Training Phase, on the Dev set, but I uploaded it to the Test Phase. Now the training phase is closed and does not accept submissions. Can we continue anyway?
Posted by: bozzz @ March 19, 2020, 9:32 a.m.I extended the development phase to never ends.
Posted by: benf @ March 20, 2020, 1:59 p.m.Hi,
Sorry I didn't got the information. Is the deadline for submission of test phase 24th March 2020 or have been pushed to later? if yes till when?
Posted by: subh123 @ March 20, 2020, 7:32 p.m.Hi,
Deadline was pushed to later:
We, the organization committee, agreed to extend the competition until the 31. March, sorry for the delayed answer!
Cheers,
Fernando