Comparison Identification

Organized by varvara - Current server time: Nov. 30, 2020, 6:18 p.m. UTC

Previous

Development
Nov. 20, 2020, midnight UTC

Current

Post-evaluation
Dec. 4, 2020, midnight UTC

Next

Post-evaluation
Dec. 4, 2020, midnight UTC

Welcome!

This is the competition for the Assignment II for Statistical NLP course at CDISE, Skoltech. Your task is to train a sequence labelling model to recognise objects, aspects, and predicates in comparative sentences, i.e. sentences containing comparisons.


 Examples of sentences

Postgres is easier to install and maintain than Oracle.

[Postgres OBJECT] is [easier PREDICATE] to [install ASPECT] and [maintain ASPECT] than [Oracle OBJECT].

Instances can be multiword:

Advil works better for body aches and pains than Motrin.

[Advil OBJECT] works [better PREDICATE] for [body aches ASPECT] and [pains ASPECT] than [Motrin OBJECT].


Data format

The provided data files are in CoNLL format. Each line contains one word and its label, separated by a tab ("Word<TAB>label"), the end of sentence is marked with an empty line. The labels are in BIO format, where each of the entity labels ("Object", "Aspect", "Predicate") is prepended with a prefix "B-" or "I-", indicating the beginning of an entity (the first word of an entity) and the inside of an entity (the second and all subsequent words). Words which are not a part of an entity are labelled with "O":

advil    B-Object
works    O
better    B-Predicate
for        O
body    B-Aspect
aches    I-Aspect
and        O
pains    B-Aspect
than    O
motrin    B-Object
.        O


Submission

Your system should produce labelling in the same CoNLL format as the provided training data. You should submit file in this format with ".tsv" file extension and packed in a zip archive. The names of the file and the archive don't matter.

In order to submit you should switch to "Participate" tab and choose "Submit / View results" field. You will see the "Submit" button. After you have uploaded your file, you will see it in the list of your submissions along with its current status ("Submitted", "Running", "Finished", etc.). If the submission status is "Finished", it means that the evaluation has been completed successfully. In that case you can add this submission to the public leaderboard by pressing the button "Add to leaderboard". Only one of your submissions can be in the leaderboard at the same time. We suggest putting the best one to the leaderboard.

In the first phase you should label the provided development set.

In the second phase you should label the test set. Please note that we provide ground truth labels for the development set in order to give you a possibility to analyse the errors of your model. However, the test set will be hidden to prevent cheating.


 

Troubleshooting

If your submission failed, please take a look at the scoring error log. It might happen because of the wrong format of your data (unknown labels, non-matching number of sentences between the reference and your submission, etc.). If you cannot figure out the reason of failure, please contact the TAs.

If your submission has the status "Submitted" or "Running" for a long time, it means that CodaLab has a lot of jobs, and it takes a long time to process your submission. In that case please keep calm and don't rush to resubmit. It typically runs the submissions in at most several hours.

Evaluation Criteria

The primary evaluation metric will be the average F1-score for the three classes of entities - Objects, Aspects, Predicates (referred to as F1 Average in the leaderboard). We also report F1-scores for the individual classes. We use the F1-score implementation from the package seqeval. For the average F1-score, you should call the f1_score function with the parameter average="weighted" and for F1-scores for the individual classes use average=None.

In addition to that, we report a secondary set of metrics, namely, F1-scores which take into account partially matching entities. These metrics are referred to in the leaderboard as partial. They are in-house modifications of seqeval F1-score and are always equal or greater than the corresponding standard F1-scores. These scores will be used for algorithm analysis purposes or for defining winning models in case of ties.

Terms and Conditions

This is the task for the course Assignment. Scores of the models you submit will influence your overall course score. Therefore, we kindly ask you to accomplish the task on your own. Please do not use solutions of your classmates and do not cheat in any other ways. If you find any data leakage, please inform the lecturer or teacher assistants.

Development

Start: Nov. 20, 2020, midnight

Final test

Start: Nov. 27, 2020, midnight

Post-evaluation

Start: Dec. 4, 2020, midnight

Competition Ends

Never

You must be logged in to participate in competitions.

Sign In
# Username Score
1 addmitriev 0.777308
2 palette_knife25 0.773742
3 varvara 0.072352