CLEF ProtestNews 2019: Evaluation

 LEADERBOARD LINKS :

Screenshot1 

Screenshot2 

The lab aims at evaluating generalizability of text classification and information extraction tools. Therefore, we designed the evaluation as follows. The training data is obtained from a single country, which is the source country. The evaluation is consists of two steps. The first step of evaluation, which we call Test 1 or intermediate evaluation, is performed on data from the source country. The second step of evaluation, which we call Test 2 or Final evaluation, is performed on data from a target country, which is China in our setting. The performance metrics for both Test 1 and Test 2 are described below.

FINAL EVALUATION PHASE

In addition to the News articles obtained from India, we will make use of News Articles from China to make the final evaluation of the participants for all three tasks.

Our main aim in choosing this approach is to favor models which can generalize better and adapt to new domains better.

CLEF ProtestNews 2019: Terms and Conditions

The participants of the competition are assumed to have read and agreed the terms and conditions listed below.

• The data shared for the competition must only be used for research related purposes.
• Parts of the data provided can be shared by participants for illustrative purposes only. Sharing the datasets in a way that makes it possible to reconstruct the whole dataset is not allowed.
• Only individuals/teams that have approved registration to the competition has the rights to make use of the data. It is the responsibility of each participant to prevent the third parties to access the data.
• Copyright holders of the datasets shared for the competition retain all rights regarding the use and distribution of all the material.

ProtestNews 2019 Organizing Committee

CLEF ProtestNews 2019: Submission

We provide tbe data for each task in subfolder of their own. You should have gotten the data from one of the organizers manually.

 Task 1 and 2

For task 1 and 2 we give "test.json" and "china_test.json" as test files. Each of them have a "prediction" field in them with null values. For your submissions you must fill the "prediction" field with your model's predictions (please provide integers) and submit these two files in a zip file.

Rules for submission:

• Submitted files in zip must have the same names as the provided test files. ("test.json", "china_test.json")
• The zip file must not contain a folder, only these two files.
• If you do not provide any of the test files in the zip file, its corresponding score will be 0 and will also affect the average score.

Task 3

We will provide a token-per-line format data. The participants must make their predictions with a tab between the token and the prediction at each line.

For task 1 and 2 we give "test.txt" and "china_test.txt" as test files. Each of them only have tokens in them unlike train and dev files. For your submissions you must add your model's predictions for each token (seperating the predictions and tokens with tab like in train and dev data) and submit these two files in a zip file.

Rules for submission:

• Submitted files in zip must have the same names as the provided test files. ("test.txt", "china_test.txt")
• The zip file must not contain a folder, only these two files.
• If you do not provide any of the test files in the zip file, its corresponding score will be 0 and will also affect the average score.
• All lines must exactly match with the provided test file. Otherwise the scoring program will not be able to calculate the score.
• Be careful with extra empty lines and assure that each token at each line and empty lines completely overlap.
• You can check out here for a sample submission.

Tasks

Task 1 : The task is to classify news documents as protest (1) or non-protest (0), given the raw document. 

Task 2 : The task is to classify sentences as a sentence containing an event-trigger (1) or not (0), given the sentence and the news article containing that sentnece.

Task 3 : The task is to extract various information from a given event sentence such as location, time and participant of an event. 

Task 1 and Task 2

Both Task 1 and Task 2 are binary classification tasks.

The submission will be evaluated using the F₁ score for Task 1 and Task 2. 

We will give intermediate results at the end of the first phase several times in order to ensure that competitors can get feedback about their models.

For intermediate evaluation of Task 1 and Task 2 we will use news articles from India. For final evaluation we use a mixture of news articles from china and india. 

Final evaluation will be made on the test set that will be released for the final phase.

Task 3

For Task 3, F1 metric will be used. BIO tagging scheme is used to annotate the corpus for various information types.

We will provide intermediate results under the Task 3 phase for Task 3 submissions.

The final results will be given on the test set which will be provided later.