Competition

Scoring

Our scoring considers classification accuracy and evidence recall.

• We will only award points for accuracy if the correct evidence is found.
• For a claim, we consider the correct evidence to be found if the complete set of annotated sentences is returned.

Baseline system

For a detailed description of the data annotation process and baseline results see the paper

Data Format

The data will be distributed in JSONL format with one example per line (see http://jsonlines.org/ for more details).

In addition to the task-specific dataset, the full set of Wikipedia pages (segmented at the sentence level) will be distributed on the data tab or on our website https://sheffieldnlp.github.io/fever.

Training/Development Data format

The training and development data will contain 4 fields:

• id: The ID of the claim
• label: The annotated label for the claim. Can be one of SUPPORTS|REFUTES|NOT ENOUGH INFO.
• claim: The text of the claim.
• evidence: A list of evidence sets (lists of [Annotation ID, Evidence ID, Wikipedia URL, sentence ID] tuples) or a [Annotation ID, Evidence ID, null, null] tuple if the label is NOT ENOUGH INFO.

Below are examples of the data structures for each of the three labels.

Supports Example

{
    "id": 62037,
    "label": "SUPPORTS",
    "claim": "Oliver Reed was a film actor.",
    "evidence": [
        [
            [<annotation_id>, <evidence_id>, "Oliver_Reed", 0]
        ],
        [
            [<annotation_id>, <evidence_id>, "Oliver_Reed", 3],
            [<annotation_id>, <evidence_id>, "Gladiator_-LRB-2000_film-RRB-", 0]
        ],
        [
            [<annotation_id>, <evidence_id>, "Oliver_Reed", 2],
            [<annotation_id>, <evidence_id>, "Castaway_-LRB-film-RRB-", 0]
        ],
        [
            [<annotation_id>, <evidence_id>, "Oliver_Reed", 1]
        ],
        [
            [<annotation_id>, <evidence_id>, "Oliver_Reed", 6]
        ]
    ]
}

Refutes Example

{
    "id": 78526,
    "label": "REFUTES",
    "claim": "Lorelai Gilmore's father is named Robert.",
    "evidence": [
        [
            [<annotation_id>, <evidence_id>, "Lorelai_Gilmore", 3]
        ]
    ]
}

NotEnoughInfo Example

{
    "id": 137637,
    "label": "NOT ENOUGH INFO",
    "claim": "Henri Christophe is recognized for building a palace in Milot.",
    "evidence": [
        [
            [<annotation_id>, <evidence_id>, null, null]
        ]
    ]
}

Test Data format

The test data will follow the same format as the training/development examples, with the label and evidence fields removed.

{
    "id": 78526,
    "claim": "Lorelai Gilmore's father is named Robert."
}

Answer Submission Instructions

• Go to Codalab
• Create a team/system account
• Submit answers file as a ZIP file containing a single predictions file (file name must be predictions.jsonl). One claim object per line submitting predicted evidence as a set of [Page, Line ID] tuples. Each json object should adhere to the following format (with line breaks removed) and the order of the claims must be preserved.

{
    "id": 78526,
    "predicted_label": "REFUTES",
    "predicted_evidence": [
        ["Lorelai_Gilmore", 3]
    ]
}

Evaluation

In the first instance, submissions will be measured on the correctness of the label assignment for claims. If a claim is labelled as supported or refuted, additionally the evidences will be checked against the list of annotated evidences. The label will be considered as correct if atleast one provided evidence matches the annotated evidences.

At a later date (before the workshop) we will manually assess evidence marked as false-positive and release updated scores and an update to the corpus.

Best practice

1. We highly discourage the usage of submission server for parameter tuning. Such submissions will be disqualified from the task.
2. If you are using any additional data beyond wikipedia to build your system, please mention it clearly in your submission notes.

BibTeX

@inproceedings{Thorne18Fever,
    author = {Thorne, James and Vlachos, Andreas and Christodoulopoulos, Christos and Mittal, Arpit},
    title = {{FEVER}: a Large-scale Dataset for Fact Extraction and VERification},
    booktitle = {NAACL-HLT},
    year = {2018}
}

License