Competition

Evaluation Criteria

We will use F1 score as our evaluation metric, following our paper in ACL 2017.

We will provide the input data for each subtask, and the participants will need to make prediction based on the given input data. Name your output files for SubTask 1, 2, 3, and 4 as Task1.out, Task2.out, Task3,out, and Task4.out respectively. The Codalab submission format requires the submission to be in a single zip file containing the output files. We provided a sample 'submission.zip' file containing the benchmark outputs as an example of what it looks like.

SubTask 1

We will provide a list of sentences, the participants need to predict whether the sentences are relevant for inferring the malware's actions and capabilities.

For each sentences, the participants will need to output 1 if the sentence is relevant or 0 if it is irrelevant.

SubTask 2

Using the same sentences as provided in Task 1, the participants need to predict the token labels in the sentences. The output needs to be in BIO format. There are 3 types of token labels: "Action", "Entity", and "Modifier". 

SubTask 3

We will provide a different set of sentences and their token labels. They will come in separate files according to their source document. The participants need to predict the relations between the token labels. This task will be treated as a binary classification task. For each entity pair, the participant need to output O if there is no relation between the entities, or <relation_type> if there is a relation between them. The relation types are: "SubjAction", "ActionObj", "ActionMod", and "ModObj". Token 0 will be reserved for the root. Any other tokens without a parent will be connected to this root entity with the 'ROOT' relation. 

Here's an example output for SubTask 3:

SubTask 4

We will provide a different set of sentences, with annotations of their token labels and their relation labels. The sentences will be separated into several files based on the document it came from. The participants need to predict the attributes for each 'Action' token in 4 categories: 'ActionName', 'Capability', 'StrategicObjectives', and 'TacticalObjectives'. Output 'O' if the 'Action' token doesn't correspond to any attribute in that category. The detailed list of attributes can be found in the MalwareTextDB annotation guideline. 

The annotations given will follow the brat annotation format. Each line of annotation will be given an ID depending on which type is it. The annotation for token labels starts with 'T', the annotation for relation labels starts with 'R', and the annotation for attribute labels starts with 'A'. Please refer to the linked webpage for a more detailed explanation. We did not include the attribute labels for the input files, since it is what the participants need to predict.

Output format (separated by tab):

<doc_id> <action_label> <action_name> <capability> <strategic_objectives> <tactical_objectives>

Here's an example of what the output looks like:

For Subtask 3 and 4, we will provide a file called 'doclist.txt' which contains the file names. The doc_id will be the row number of that document in the 'doclist.txt' starting from index 0.

For example, given a doclist.txt with this content:

Operation_Snowman will have the doc_id 0 and The_Monju_Incident will have the doc_id 1.

For all subtasks, the output must follow the order of appearance of the sentence in the document and the order of the document in the doclist.txt (if given). Additionally for subtask 4, the action label within each document must also be output in ascending order according to their number (T2, T8, T13, .....etc.).

Terms and Conditions

By submitting results to this competition, you consent to the public release of your scores at the SemEval-2018 workshop and in the associated proceedings, at the task organizers' discretion. Scores may include, but are not limited to, automatic and manual quantitative judgements, qualitative judgements, and such other metrics as the task organizers see fit. You accept that the ultimate decision of metric choice and score value is that of the task organizers.

You further agree that the task organizers are under no obligation to release scores and that scores may be withheld if it is the task organizers' judgement that the submission was incomplete, erroneous, deceptive, or violated the letter or spirit of the competition's rules. Inclusion of a submission's scores is not an endorsement of a team or individual's submission, system, or science.

You further agree that your system may be named according to the team name provided at the time of submission, or to a suitable shorthand as determined by the task organizers.

You agree not to redistribute the test data except in the manner prescribed by its licence.