SemEval-2018 Task 1: Affect in Tweets

Organized by felipebravom - Current server time: Nov. 22, 2017, 11:46 p.m. UTC

Current

Pre-Evaluation Period
Aug. 14, 2017, midnight UTC

Next

Evaluation Period
Jan. 8, 2018, midnight UTC

SemEval-2018 Task 1: Affect in Tweets

SemEval-2018: International Workshop on Semantic Evaluation will be held in conjunction with one of the top NLP conferences in 2018 (ACL/NAACL/EMNLP).

Join the official task mailing group: 
EmotionIntensity@googlegroups.com

Background and Significance: We use language to communicate not only the emotion or sentiment we are feeling but also the intensity of the emotion or sentiment. For example, our utterances can convey that we are very angry, slightly sad, absolutely elated, etc. Here, intensity refers to the degree or amount of an emotion or degree of sentiment. We will refer to emotion-related categories such as anger, fear, sentiment, and arousal, by the term affect. Existing affect datasets are mainly annotated categorically without an indication of intensity. Further, past shared tasks have almost always been framed as classification tasks (identify one among n affect categories for this sentence). In contrast, it is often useful for applications to know the degree to which affect is expressed in text.

Tasks: We present an array of tasks where systems have to automatically determine the intensity of emotions (E) and intensity of sentiment (aka valence V) of the tweeters from their tweets. (The term tweeter refers to the person who has posted the tweet.) We also include a multi-label emotion classification task for tweets. For each task, we provide separate training and test datasets for English, Arabic, and Spanish tweets. The individual tasks are described below:

  1. EI-reg (an emotion intensity regression task): Given a tweet and an emotion E, determine the  intensity of E that best represents the mental state of the tweeter—a real-valued score between 0 (least E) and 1 (most E).
    • Separate datasets are provided for anger, fear, joy, and sadness.

  2. EI-oc (an emotion intensity ordinal classification task): Given a tweet and an emotion E, classify the tweet into one of four ordinal classes of intensity of E that best represents the mental state of the tweeter.
    • Separate datasets are provided for anger, fear, joy, and sadness.

  3. V-reg (a sentiment intensity regression task): Given a tweet, determine the intensity of sentiment or valence (V) that best represents the mental state of the tweeter—a real-valued score between 0 (most negative) and 1 (most positive).

  4. V-oc (a sentiment analysis, ordinal classification, task): Given a tweet, classify it into one of seven ordinal classes, corresponding to various levels of positive and negative sentiment intensity, that best represents the mental state of the tweeter.

  5. E-c (an emotion classification task): Given a tweet, classify it as 'neutral or no emotion' or as one, or more, of eleven given emotions that best represent the mental state of the tweeter.

Here, E refers to emotion, EI refers to emotion intensity, V refers to valence or sentiment intensity, reg refers to regression, oc refers to ordinal classification, c refers to classification. 

Together, these tasks encompass various emotion and sentiment analysis tasks. You are free to participate in any number of tasks and on any of the datasets. Further details on each of the tasks are provided below.


1. Task EI-reg: Detecting Emotion Intensity (regression)

Given:

  • a tweet

  • an emotion E (anger, fear, joy, or sadness)

Task: determine the  intensity of E that best represents the mental state of the tweeter—a real-valued score between 0 and 1:

  • a score of 1: highest amount of E can be inferred

  • a score of 0: lowest amount of E can be inferred

For each language: 4 training sets and 4 test sets: one for each emotion E.

(Note that the absolute scores have no inherent meaning -- they are used only as a means to convey that the instances with higher scores correspond to a greater degree of E than instances with lower scores.)


2. Task EI-oc: Detecting Emotion Intensity (ordinal classification)

Given:

  • a tweet

  • an emotion E (anger, fear, joy, or sadness)

Task: classify the tweet into one of four ordinal classes of intensity of E that best represents the mental state of the tweeter:

  • 0: no E can be inferred

  • 1: low amount of E can be inferred

  • 2: moderate amount of E can be inferred

  • 3: high amount of E can be inferred

For each language: 4 training sets and 4 test sets: one for each emotion E.


3. Task V-reg: Detecting Valence or Sentiment Intensity (regression)

Given:

  • a tweet

Task: determine the intensity of sentiment or valence (V) that best represents the mental state of the tweeter—a real-valued score between 0 and 1:

  • a score of 1: most positive mental state can be inferred

  • a score of 0: most negative mental state can be inferred

For each language: 1 training set, 1 test set.

(Note that the absolute scores have no inherent meaning -- they are used only as a means to convey that the instances with higher scores correspond to a greater degree of positive sentiment than instances with lower scores.) 


4. Task V-oc: Detecting Valence (ordinal classification) -- This is the traditional Sentiment Analysis Task

Given:

  • a tweet

Task: classify the tweet into one of seven ordinal classes, corresponding to various levels of positive and negative sentiment intensity, that best represents the mental state of the tweeter:

  • 3: very positive mental state can be inferred

  • 2: moderately positive mental state can be inferred

  • 1: slightly positive mental state can be inferred

  • 0: neutral or mixed mental state can be inferred

  • -1: slightly negative mental state can be inferred

  • -2: moderately negative mental state can be inferred

  • -3: very negative mental state can be inferred

For each language: 1 training set, 1 test set. 


5. Task E-c: Detecting Emotions (multi-label classification) -- This is a traditional Emotion Classification Task

Given:

  • a tweet

Task: classify the tweet as 'neutral or no emotion' or as one, or more, of eleven given emotions that best represent the mental state of the tweeter:

  • anger (also includes annoyance and rage) can be inferred
  • anticipation (also includes interest and vigilance) can be inferred
  • disgust (also includes disinterest, dislike and loathing) can be inferred
  • fear (also includes apprehension, anxiety, concern, and terror) can be inferred
  • joy (also includes serenity and ecstasy) can be inferred
  • love (also includes affection) can be inferred
  • optimism (also includes hopefulness and confidence) can be inferred
  • pessimism (also includes cynicism and lack of confidence) can be inferred
  • sadness (also includes pensiveness and grief) can be inferred
  • suprise (also includes distraction and amazement) can be inferred
  • trust (also includes acceptance, liking, and admiration) can be inferred


For each language: 1 training set, 1 test set.

(Note that the set of emotions includes the eight basic emotions as per Plutchik (1980), as well as a few other emotions that are common in tweets (love, optimism, and pessimism).)


Paper: Participants will be given the opportunity to write a system-description paper that describes their system, resources used, results, and analysis. This paper will be part of the official SemEval-2018 proceedings. The paper is to be four pages long plus two pages at most for references. The papers are to follow the format and style files provided by ACL/NAACL/EMNLP-2018.

Related Past Shared Tasks on Affect Intensity

EVALUATION

 

For the Tasks EI-reg, EI-oc, V-reg, and V-oc 

 

Official Competition Metric: For each task, language, and affect category, systems are evaluated by calculating the Pearson Correlation Coefficient with the Gold ratings/labels.

  • The correlation scores across all four emotions will be averaged (macro-average) to determine the bottom-line competition metric for EI-reg and EI-oc by which the submissions will be ranked for those tasks.

  • The correlation scores for valence will be used as the bottom-line competition metric for V-reg and V-oc by which the submissions will be ranked for those tasks.


The evaluation script for the EI-reg subtask (which also acts as a format checker) is available here. The full official evaluation script that covers all subtasks will be available soon. As and when the training, development, and test sets are made available, you should run the script on your system’s predictions on that data for purposes such as cross-validation experiments, determining progress on the development set, and to check the format of your submission.

The CodaLab website for the 2017 task is still open. You can train on the official 2017 training data and test on the official 2017 test set and compare against the best 2017 systems on the Leaderboard.

Secondary Evaluation Metrics: Apart from the official competition metric described above, some additional metrics will also be calculated for your submissions. These are intended to provide a different perspective on the results. They will not be shown on the official leaderboard, but will be provided via an output file for each submission.

The secondary metric used for the regression tasks:

  • Pearson correlation for a subset of the test set that includes only those tweets with intensity score greater or equal to 0.5.

The secondary metrics used for the ordinal classification tasks:

  • Pearson correlation for a subset of the test set that includes only those tweets with intensity classes low X, moderate X, or high X (where X is an emotion). We will refer to this set of tweets as the some-emotion subset
  • Weighted quadratic kappa on the full test set.
  • Weighted quadratic kappa on the some-emotion subset of the test set.

 

For the Task E-c 

 

Official Competition Metric: For each language, systems are evaluated by calculating multi-label accuracy (or Jaccard index). Since this is a multi-label classification task, each tweet can have one or more gold emotion labels, and one or more predicted emotion labels. Multi-label accuracy is defined as the size of the intersection of the predicted and gold label sets divided by the size of their union. This measure is calculated for each tweet t, and then is averaged over all tweets in the dataset T:

                

where Gt is the set of the gold labels for tweet t, Pt is the set of the predicted labels for tweet t, and T is the set of tweets.

The official evaluation script will be released shortly.

Secondary Evaluation Metrics: Apart from the official competition metric (multi-label accuracy), we will also calculate micro-averaged F-score and macro-averaged F-score for your submissions. These additional metrics are intended to provide a different perspective on the results. They will not be shown on the official leaderboard, but will be provided via an output file for each submission.


Micro-averaged F-score is calculated as follows:

where E is the given set of eleven emotions.

 

Macro-averaged F-score is calculated as follows:

     

Terms and Conditions

  • A participant can be involved in exactly one team (no more). If there are reasons why it makes sense for you to be on more than one team, then email us before the evaluation period begins. In special circumstances this may be allowed.

  • Each team must create and use exactly one CodaLab account.

  • Team constitution (members of a team) cannot be changed after the evaluation period has begun.

  • Each team can submit as many as ten submissions during the evaluation period. However, only the final submission will be considered as the official submission to the competition.

  • You will not be able to see results of your submission on the test set.

  • You will be able to see any warnings and errors for each of your submission.

  • Leaderboard is disabled.

  • Special situations will be considered on a case by case basis. If you have reached the limit of ten submissions and there are extenuating circumstances due to which you need to make one more submission, send us an email before the evaluation period deadline (January 28, 2018), and we will likely remove one of your earlier submissions.

  • Once the competition is over, we will release the gold labels and you will be able to determine results on various system variants you may have developed. We encourage you to report results on all of your systems (or system variants) in the system-description paper. However, we will ask you to clearly indicate the result of your official submission.

Organizers of the shared task:

 

    Saif M. Mohammad

    saif.mohammad@nrc-cnrc.gc.ca

    National Research Council Canada

 

    Felipe Bravo-Marquez

    fbravoma@waikato.ac.nz

    The University of Waikato

 

    Mohammad Salameh

    msalameh@qatar.cmu.edu

    Carnegie Mellon University, Qatar

 

    Svetlana Kiritchenko

    svetlana.kiritchenko@nrc­-cnrc.gc.ca

    National Research Council Canada

DATA

 

If you use any of the data below, please cite this paper:

 

Saif M. Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018. Semeval-2018 Task 1: Affect in tweets. In Proceedings of International Workshop on Semantic Evaluation (SemEval-2018).

 

Data: 
(Here, E refers to emotion, EI refers to emotion intensity, V refers to valence or sentiment intensity, reg refers to regression, oc refers to ordinal classification, c refers to classification.) 

  • EI-reg:

    • English (Note: This particular training set was created from a BWS annotation effort in 2016. The development and test sets were created from a common 2017 annotation effort. Thus, the scores for tweets across the training and development sets or across the training and test sets are not directly comparable. However, the scores in each dataset indicate relative positions of the tweets in that dataset.)  

    • Arabic

    • Spanish

      • Test Set

  • EI-oc:

  • V-reg:

  • V-oc:

  • E-c: (Regarding the notation in these files: For a given emotion, 1 means emotion can be inferred, whereas 0 means emotion cannot be inferred. 0's for all of the 11 emotions means 'neutral or no emotion'.)

 

Note: The datasets above share a large number of common tweets, however, they were often created from independent annotations by different annotators. Further, decisions on where to mark thresholds in the different datasets are made independently as well. For example, in E-c we chose a somewhat generous criteria that if at least two out of seven people indicate that a certain emotion can be inferred, then that emotion is chosen as one of the labels for the tweet (likely along with another emotion with 3, 4, or 5 votes). Thus, a small number of inconsistencies in the annotations across different datasets is expected. For example, a tweet may be marked as 'no anger' in E-oc, but may have 'anger' as one of its labels in E-c. Of course, such instances are greatly outnumbered by consistent annotations across the datasets.

Submission format:

A valid submission for CodaLab is a zip-compressed file with files containing the predictions made for all the subtasks you want to participate in. Submitted files must have the same format as the training and test files after replacing the NONEs in the last columns with your system's predictions.

The filenames associated with each subtask and the corresponding line formats are given below:

  • EI-reg: id[tab]tweet[tab]emotion[tab]score. Note that the emotion name must be in English even for Spanish and Arabic data

    • English

      • EI-reg_en_anger_pred.txt

      • EI-reg_en_fear_pred.txt

      • EI-reg_en_sadness_pred.txt

      • EI-reg_en_joy_pred.txt

    • Arabic

      • EI-reg_ar_anger_pred.txt

      • EI-reg_ar_fear_pred.txt

      • EI-reg_ar_sadness_pred.txt

      • EI-reg_ar_joy_pred.txt

    • Spanish

      • EI-reg_es_anger_pred.txt

      • EI-reg_es_fear_pred.txt

      • EI-reg_es_sadness_pred.txt

      • EI-reg_es_joy_pred.txt

 

  • EI-oc: id[tab]tweet[tab]emotion[tab]class

    • English

      • EI-oc_en_anger_pred.txt

      • EI-oc_en_fear_pred.txt

      • EI-oc_en_sadness_pred.txt

      • EI-oc_en_joy_pred.txt

    • Arabic

      • EI-oc_ar_anger_pred.txt

      • EI-oc_ar_fear_pred.txt

      • EI-oc_ar_sadness_pred.txt

      • EI-oc_ar_joy_pred.txt

    • Spanish

      • EI-oc_es_anger_pred.txt

      • EI-oc_es_fear_pred.txt

      • EI-oc_es_sadness_pred.txt

      • EI-oc_es_joy_pred.txt

 

  • V-reg: id[tab]tweet[tab]valence[tab]score

    • English

      • V-reg_en_pred.txt

    • Arabic

      • V-reg_ar_pred.txt

    • Spanish

      • V-reg_es_pred.txt

 

  • V-oc: id[tab]tweet[tab]valence[tab]class

    • English

      • V-oc_en_pred.txt

    • Arabic

      • V-oc_ar_pred.txt

    • Spanish

      • V-oc_es_pred.txt

 

  • E-c:  id[tab]tweet[tab]anger_val[tab]anticipation_val[tab]disgust_val[tab] fear_val[tab]joy_val[tab]love_val[tab]optimism_val[tab]pessimism_val[tab] sadness_val[tab]surprise_val[tab]trust_val
    (Note: Each emotion value (e.g., love_val) takes binary values: 1 means emotion can be inferred, whereas 0 means emotion cannot be inferred. 0's for all of the 11 emotions means 'neutral or no emotion'.)

    • English

      • E-C_en_pred.txt

    • Arabic

      • E-C_ar_pred.txt

    • Spanish

      • E-C_es_pred.txt


Participants are not required to participate in all subtasks. A valid submission must provide at least all the files associated with one combination of subtask and language.

Example of a valid combination of files:

  • EI-reg_en_anger_pred.txt

  • EI-reg_en_fear_pred.txt

  • EI-reg_en_sadness_pred.txt

  • EI-reg_en_joy_pred.txt

A zip file containing the above files will only be participating in the EI-reg task for English.

Example of an invalid combination of files:

  • EI-oc_en_sadness_pred.txt

  • EI-reg_en_joy_pred.txt

  • EI-reg_es_joy_pred.txt

Schedule:

  • Training data ready: September 25, 2017

  • Evaluation period starts: January 8, 2018

  • Evaluation period ends: January 24, 2018

  • Results posted: Feb 5, 2018

  • System description paper submission deadline: TBD

  • Author notifications : TBD

  • Camera ready submissions due: TBD

Manual Annotation: Obtaining real-valued annotations has several challenges. Respondents are faced with a higher cognitive load when asked for real-valued scores as opposed to simply classifying terms into pre-chosen discrete classes. Besides, it is difficult for an annotator to remain consistent with his/her annotations. Further, the same score may map to different sentiment scores in the minds of different annotators. One could overcome these problems by providing annotators with pairs of terms and asking which is stronger in terms of association with the property of interest (a comparative approach); however, that requires a much larger set of annotations (order NxN, where N is the number of instances to be annotated).

Best–Worst Scaling (BWS), also sometimes referred to as Maximum Difference Scaling (MaxDiff), is an annotation scheme that exploits the comparative approach to annotation (Louviere and Woodworth, 1990; Cohen, 2003; Louviere et al., 2015). Annotators are given four items (4-tuple) and asked which item is the Best (highest in terms of the property of interest) and which is the Worst (least in terms of the property of interest). These annotations can then be easily converted into real-valued scores of association between the items and the property, which eventually allows for creating a ranked list of items as per their association with the property of interest.

Kiritchenko and Mohammad (2016, 2017) show that ranking of terms remains remarkably consistent even when the annotation process is repeated with a different set of annotators. See the hyperlinked webpages for details on Reliability of the Annotations and a comparison of BWS with Rating Scales.

We created all the datasets for this task using Best–Worst Scaling.

 

Papers:

You are free to build a system from scratch using any available software packages and resources, as long as they are not against the spirit of fair competition. You must report all resources used in the system-description paper.

Baseline System

In order to assist testing of ideas, we also provide the AffectiveTweets Package that you can use and build on. A common use of the package is to generate feature vectors from various resources and append it to one’s own feature representation of the tweet. The use of this package is completely optional. It is available here. Instructions for using the package are available here.

The AffectiveTweets package was used by the teams that ranked first, second, and third in the WASSA-2017 Shared Task on Emotion Intensity.


Word-Emotion and Word-Sentiment Association lexicons

Large lists of manually created and automatically generated word-emotion and word-sentiment association lexicons are available here.

References:

  • Emotion Intensities in Tweets. Saif M. Mohammad and Felipe Bravo-Marquez. In Proceedings of the sixth joint conference on lexical and computational semantics (*Sem), August 2017, Vancouver, Canada.

  • WASSA-2017 Shared Task on Emotion Intensity. Saif M. Mohammad and Felipe Bravo-Marquez. In Proceedings of the EMNLP 2017 Workshop on Computational Approaches to Subjectivity, Sentiment, and Social Media (WASSA), September 2017, Copenhagen, Denmark.

  • Picard, R. W. (1997, 2000). Affective computing. MIT press.

  • Using Hashtags to Capture Fine Emotion Categories from Tweets. Saif M. Mohammad, Svetlana Kiritchenko, Computational Intelligence, Volume 31, Issue 2, Pages 301-326, May 2015.

  • Best-Worst Scaling More Reliable than Rating Scales: A Case Study on Sentiment Intensity Annotation. Kiritchenko, S. and Mohammad, S. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL-2017), Vancouver, Canada, 2017.

  • Crowdsourcing a Word-Emotion Association Lexicon, Saif Mohammad and Peter Turney, Computational Intelligence, 29 (3), 436-465, 2013.

  • Ekman, P. (1992). An argument for basic emotions. Cognition and Emotion, 6 (3), 169-200.

  • #Emotional Tweets, Saif Mohammad, In Proceedings of the First Joint Conference on Lexical and Computational Semantics (*Sem), June 2012, Montreal, Canada.

  • Portable Features for Classifying Emotional Text, Saif Mohammad, In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, June 2012, Montreal, Canada.

  • Strapparava, C., & Mihalcea, R. (2007). Semeval-2007 task 14: Affective text. In Proceedings of SemEval-2007, pp. 70-74, Prague, Czech Republic.

  • From Once Upon a Time to Happily Ever After: Tracking Emotions in Novels and Fairy Tales, Saif Mohammad, In Proceedings of the ACL 2011 Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities (LaTeCH), June 2011, Portland, OR.

  • Plutchik, R. (1980). A general psychoevolutionary theory of emotion. Emotion: Theory, research, and experience, 1(3), 3-33.

  • Stance and Sentiment in Tweets. Saif M. Mohammad, Parinaz Sobhani, and Svetlana Kiritchenko. Special Section of the ACM Transactions on Internet Technology on Argumentation in Social Media, In Press.

  • Determining Word-Emotion Associations from Tweets by Multi-Label Classification. Felipe Bravo-Marquez, Eibe Frank, Saif Mohammad, and Bernhard Pfahringer. In Proceedings of the 2016 IEEE/WIC/ACM International Conference on Web Intelligence (WI'16), Omaha, Nebraska, USA.

  • Challenges in Sentiment Analysis. Saif M. Mohammad, A Practical Guide to Sentiment Analysis, Springer, 2016.

  • Osgood, C. E., Suci, G. J., & Tannenbaum, P. (1957). The measurement of meaning. University of Illinois Press.

  • Capturing Reliable Fine-Grained Sentiment Associations by Crowdsourcing and Best-Worst Scaling. Svetlana Kiritchenko and Saif M. Mohammad. In Proceedings of the 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. June 2016. San Diego, CA.

  • Ortony, A., Clore, G. L., & Collins, A. (1988). The Cognitive Structure of Emotions. Cambridge University Press.

  • Semeval-2016 Task 7: Determining Sentiment Intensity of English and Arabic Phrases. Svetlana Kiritchenko, Saif M. Mohammad, and Mohammad Salameh. In Proceedings of the International Workshop on Semantic Evaluation (SemEval-16). June 2016. San Diego, California.

  • Alm, C. O. (2008). Affect in text and speech. ProQuest.

  • Aman, S., & Szpakowicz, S. (2007). Identifying expressions of emotion in text. In Text, Speech and Dialogue, Vol. 4629 of Lecture Notes in Computer Science, pp. 196-205.

  • The Effect of Negators, Modals, and Degree Adverbs on Sentiment Composition. Svetlana Kiritchenko and Saif M. Mohammad, In Proceedings of the NAACL 2016 Workshop on Computational Approaches to Subjectivity, Sentiment, and Social Media (WASSA), June 2014, San Diego, California.

  • Sentiment Analysis: Detecting Valence, Emotions, and Other Affectual States from Text. Saif M. Mohammad, Emotion Measurement, 2016.

  • NRC-Canada-2014: Detecting Aspects and Sentiment in Customer Reviews, Svetlana Kiritchenko, Xiaodan Zhu, Colin Cherry, and Saif M. Mohammad. In Proceedings of the eighth international workshop on Semantic Evaluation Exercises (SemEval-2014), August 2014, Dublin, Ireland.

  • Barrett, L. F. (2006). Are emotions natural kinds?. Perspectives on psychological science, 1(1), 28-58.

FAQ

 

Q. Why restrict the number of official submissions to one?

A. Since this is a competition, we do not want teams to submit a large number of submissions using different parameters and systems without being confident which will work best.

Even though the number of official submissions is restricted to one, the gold data will be released soon after the evaluation period. Thus you can use it to determine results from many different system variants. You are strongly  encouraged to report these additional results in the system-description paper in addition to the official submission results.


Q. How do I include more than one score on the leaderboard?

A. CodaLab allows only one score on the leaderboard per user.

Directions for Participating via CodaLab
Steps:
  1. Create an account in CodaLab (https://competitions.codalab.org/). Sign in.

  2. Edit your profile appropriately. Make sure to add a team name, and enter names of team members. (Go to "Settings", and look under "Competition settings".)

  3. Read information on all the pages of the task website.

  4. Download data: training, development, and test (when released)

  5. Run your system on the data and generate a submission file.

  6. Make submissions on the development set (Phase 1).

    • Wait a few moments for the submission to execute.

    • Click on the ‘Refresh Status’ button to check status.

    • Check to make sure submission is successful:

      • System will show status as “Finished”

      • Click on ‘Download evaluation output from scoring step’ to examine the result.

      • If you choose to, you can upload the result on the leaderboard.

    • If unsuccessful, check error log, fix format issues (if any), resubmit updated zip.

    • Number of submissions allowed is restricted to 50.  

  7. Once the evaluation period begins, you can make submissions for the test set (Phase 2). The procedure is similar to that on the dev set. These differences apply:

    • The leader board will be disabled until the end of the evaluation period.

    • You cannot see the results of your submission. They will be posted on a later date after the evaluation period ends.

    • You can still see if your submission was successful or resulted in some error.

    • In case of error, you can view the error log.

    • Number of submissions allowed is restricted to 10. However, only your final valid submission will be your official submission to the competition.

Pre-Evaluation Period

Start: Aug. 14, 2017, midnight

Evaluation Period

Start: Jan. 8, 2018, midnight

Post-Evaluation Period

Start: Jan. 24, 2018, midnight

Competition Ends

Never

You must be logged in to participate in competitions.

Sign In

Top Three

Rank Username Score
No data