Competition

Data

Our dataset covers two different domains: newswire and notes on colon cancer. The data consists annotated documents from the TimeBank/AQUAINT corpus of news articles, and documents from the THYME corpus of clinical notes.

Participants interested in the clinical notes portion of the evaluation will have to sign a data use agreement with the Mayo Clinic to obtain the raw text of the clinical notes and pathology reports (since the THYME corpus contains incompletely de-identified clinical data; the time expressions were retained). Participants of Clinical TempEval 2015, 2016, or 2017 have already completed this process and would not have to do anything more to participate in the clinical portion of Parsing Time Normalizations. New participants can follow the instructions for the process.

Please apply for a data use agreement as soon as possible! The process may take some time.

Read the DUA carefully before agreeing to it. Among other things, you will be agreeing:

• to keep the data secure using restricted passwords and encryption (e.g., on a secure server, not on personal computers)
• not to attempt to re-identify the data
• not to redistribute the data to anyone else for any purpose

The annotation is in Anafora XML format. This means that for each file in the corpus, there will be a directory. That directory will contain a XML file. The XML file contains stand-off annotations that follows the guidelines for the proposed time operator annotation scheme (example). 

Evaluation metrics

For both Track 1 (Parsing) and Track 2 (Intervals), the results will be given in terms of precision, recall and f-measure. For Track 2, our scorer includes an interpreter that can produce time intervals reading the annotations in both Anafora and TimeML formats. The scores for each track are calculated as follows: 

• Track 1:  In this track, we follow a traditional information extraction evaluation and measure the precision and recall of finding and linking the various time operators. A predicted annotation is considered as equal to the gold-standard if it has the same character span (offsets), type, and properties (with the definition applying recursively for properties that point to other annotations).
• Track 2: For this track, we evaluate the accuracy of systems with respect to the timeline. One predicted interval is evaluated against another in the gold standard if their spans overlap. The scorer finds the intersection between the predicted and the gold standard intervals. Then, the precision is calculated by dividing the length of the intersection by the length of the predicted interval. Similarly, the recall is calculated by dividing the length of the intersection by the length of the gold standard interval. The final precision and recall will be obtained by averaging all the pairs evaluated.

System Output Format

For Track 1, your system must produce Anafora XML format files. In the case of Track 2, the format of submissions can be TimeML. In any case, your directory structure should follow the following organization:

Make sure that you comply with following rules when you create  your output directory:

• The root must contain only the domain directories, namely, "Newswire" and "Cancer".
• If you don't produce output for any of the domains, do not include its corresponding directory.
• Each domain directory must contain only the corresponding document directories and their names must be same as in the dataset.
• If you don't produce output for any of the documents, do not include its corresponding directory.
• Each document directory must contain only the corresponding annotation file. 
• The name of the annotation files must match the document name and they must include a "xml" extension, for Anafora format files,  or "tml" extension, for TimeML format files.
• All the annotation files must have the same extension.

Terms and conditions

By submitting results to this competition, you consent to the public release of your scores at the SemEval-2018 workshop and in the associated proceedings, at the task organizers' discretion. Scores may include, but are not limited to, automatic and manual quantitative judgements, qualitative judgements, and such other metrics as the task organizers see fit. You accept that the ultimate decision of metric choice and score value is that of the task organizers.

You further agree that the task organizers are under no obligation to release scores and that scores may be withheld if it is the task organizers' judgement that the submission was incomplete, erroneous, deceptive, or violated the letter or spirit of the competition's rules. Inclusion of a submission's scores is not an endorsement of a team or individual's submission, system, or science.

You further agree that your system may be named according to the team name provided at the time of submission, or to a suitable shorthand as determined by the task organizers.

You agree not to redistribute the test data except in the manner prescribed by its licence.