NADI Shared Task (Subtask 1)

Organized by chiyu94 - Current server time: Nov. 30, 2020, 5:47 p.m. UTC

Previous

Test
March 31, 2020, noon UTC

Current

Post-Evaluation
June 6, 2020, noon UTC

End

Competition Ends
Never

Welcome to subtask 1 of NADI shared task!

Arabic has a widely varying collection of dialects. Many of these dialects remain under-studied due to the rarity of resources. The goal of the shared task is to alleviate this bottleneck in the context of fine-grained Arabic dialect identification. Dialect identification is the task of automatically detecting the source variety of a given text or speech segment. Previous work on Arabic dialect identification has focused on coarse-grained regional varieties such as Gulf or Levantine (e.g., Zaidan and Callison-Burch, 2013; Elfardy and Diab, 2013; Elaraby and Abdul-Mageed, 2018) or country-level varieties (e.g., Bouamor et al., 2018; Zhang and Abdul-Mageed, 2019) such as the MADAR shared task in WANLP 2019 (Bouamor, Hassan, and Habash, 2019). The MADAR shared task also involved city-level classification on human translated data. (To receive access to the data, teams intending to participate are invited to fill in the form on the official website of NADI shared task. ).

Shared Task:

The Nuanced Arabic Dialect Identification (NADI) shared task targets province-level dialects, and as such will be the first to focus on naturally-occurring fine-grained dialect at the sub-country level. The data covers a total of 100 provinces from all 21 Arab countries and comes from the Twitter domain. Evaluation and task set up follow the MADAR 2019 shared task. The subtasks involved include:

  • Subtask 1: Country-level dialect identification: A total of 21,000 tweets, covering all 21 Arab countries. This is a new dataset created for this shared task.
  • Subtask 2: Province-level dialect identification. A total of 21,000 tweets, covering 100 provinces from all 21 Arab countries. This is the same dataset as in Subtask 1, but with province labels.  

Unlabeled data: 

Participants will also be provided with an additional 10M unlabeled tweets that can be used in developing their systems for either or both of the tasks.

Metrics:

The evaluation metrics will include precision/recall/f-score/accuracy. Macro Averaged F-score will be the official metric.

Participating teams will be provided with a common training data set and a common development set. No external manually labelled data sets are allowed. A blind test data set will be used to evaluate the output of the participating teams. All teams are required to report on the development and test set in their writeups.

Important date

  • December 1, 2019: First announcement of the shared task
  • January 30, 2020: Release of training data and scoring script
  • April 30, 2020: Registration deadline
  • March 30, 2020: Test set made available 
  • June 5, 2020: Codalab shared task submission deadline
  • July 10: Shared task system paper submissions due
  • July 30: Notification of acceptance
  • September 7: Camera-ready version of shared task system papers due
  • December 12, 2020: Workshop Dates

Contact:

Please visit the official website of the NADI shared task for more information.

For any questions related to this task, please contact the organizers directly using the following email address: ubc.nadi2020@gmail.com 

 

Evaluation Criteria

Metrics: The evaluation metrics will include precision/recall/f-score/accuracy. Macro Averaged F-score will be the official metric.

Terms and Conditions

To receive access to the data, teams intending to participate are invited to fill in the form on the official website

Copyright (c) 2020 The University of British Columbia, Canada; Carnegie Mellon University Qatar; New York University Abu Dhabi. All rights reserved.

Development

Start: Jan. 30, 2020, midnight

Description: Development phase: Develop your models and submit prediction labels on the DEV set of subtask 1. Note: The name of your submission should be 'teamname_subtask1_dev_numberOFsubmission.zip' that includes a text file of your prediction (e.g., A submission 'UBC_subtask1_dev_1.zip' that is the zip file of my first prediction for subtask 1 on DEV set containing 'UBC_subtask1_dev_1.txt' file.)

Test

Start: March 31, 2020, noon

Description: Test phase: Submit your prediction labels on the TEST set of subtask 1. Each team is allowed a maximum of 3 submissions. Note: The name of your submission should be 'teamname_subtask1_test_numberOFsubmission.zip' that includes a text file of your predictions (e.g., A submission 'UBC_subtask1_test_1.zip' that is the zip file of my prediction for subtask 1 on TEST set containing the 'UBC_subtask1_test_1.txt' file.)

Post-Evaluation

Start: June 6, 2020, noon

Description: Post-Evaluation: Submit your prediction on the TEST set of subtask 1 after the competition deadline. The name of your submission should be 'teamname_subtask1_test_numberOFsubmission.zip' that includes a text file of your predictions (e.g., A submission 'UBC_subtask1_test_1.zip' that is the zip file of my prediction for subtask 1 on TEST set containing the 'ubc_subtask1_test_1.txt' file.)

Competition Ends

Never

You must be logged in to participate in competitions.

Sign In
# Username Score
1 mutaz 0.2773
2 ArabicProcessors 0.2307
3 wissam 0.2251