ML 2020 Competition

Organized by YoussefYN - Current server time: Nov. 30, 2020, 4:50 p.m. UTC

Current

Phase 1. Development
Nov. 15, 2020, midnight UTC

Next

Phase 2. Evaluation
Dec. 14, 2020, midnight UTC

End

Competition Ends
Dec. 14, 2020, 11 p.m. UTC

Welcome!

Welcome to the official page of the Project module of the course on Introduction to Machine Learning taught to the 1st year Master's students of the Innopolis University. The project will be run in the form of a competition. Details are as follows. 

Problem:

The problem that you will solve here is object recognition for 9 objects:  {airplane, car, bird, cat, deer, dog, horse, ship, truck}. 

You will build a model that takes as input a batch of RGB images of objects with shape (32, 32, 3). Note that the test sets will be of shape (n_samples, 32, 32, 3).


In this competition, there will be two domains to which the data belong. This means there are two different probability distributions from which the images were sampled. We will denote them by Ps(x, y) and Pt(x, y). Look at the Figures below to preview samples from both domains:

Images from Domain Ps(x, y):



Images from Domain Pt(x, y):


We are interested in the generalization of your model, so your model should perform well on both domains over the unseen/hidden datasets. For that purpose we have divided our dataset into 4 sets:

  • (Xs, Ys), this represents the labeled training set where (Xs, Ys) ~ Ps(x, y). This will be accessible to you.
  • (Xt), this is an unlabeled training set where Xt ~ Pt(x) -i.e., the marginal pdf of x in Pt(x, y)-. This also will be accessible to you.
  • Public test set which is hidden from you. This will be used mainly in the first phase.
  • Private test set which is also hidden and will be used in the Evaluation phase.

There's an option, not to use the unlabeled dataset, however, your score may suffer. It is your job to figure out a way to use the given unlabeled dataset (Xt) to adapt your model to the other domain. We will support you with related topics and some references that may guide you.

The competition is divided into two phases:

  1. Development phase: You will develop your model then train it on the training set. After training, you will test your model on the public test set by submitting the model following the instructions given in the "Instructions" menu. This would give you the score for your model on the public test set. You can use this score to tune your model again on the train set. You can repeat this process 100 times in total (no more than 10 times per day) until the final submission. For submission instructions, please visit the Instructions menu.

  2. Evaluation phase: After the submission deadline of the first phase is over, the submission system will stop taking any further submissions. We will take your last model on the leaderboard and evaluate it on the private test set and get the true performance of your model on this dataset that wasn't seen/revealed before to your model. Both the score and the rank among the total number of students on this dataset (private test set) will be used to calculate your grade. This will help us to assess the generalization of your model. You may read further on this in the Evaluation menu.

 

Evaluation Criteria

  • The accuracy metric is used in the evaluation.
  • The first phase of the competition will be evaluated on the public test set.
  • The second phase of the competition will be offline, where we will evaluate your last model on another dataset (private test set).

So, the first phase is for you to tune your model, and the second phase for us to evaluate it.

The final grade will be a combination of your score and your rank:

Let your score on the private test set be a ∈ [0, 1] and your rank is r  [1, S], while S is the number of participants or students.

So, there are 50 students and:

  • if your score is 0.75 and your rank is the 10th, then the grade will be = 0.75 + (1 - (9/50)) * (1 - 0.75) = 0.955 = 95.5%

  • if your score is 0.85 and your rank is the 40th, then the grade will be = 0.85 + (1 - (39/50)) * (1 - 0.85) = 0.883 = 88.3%

Notes

  • Cheating is a serious academic offense and will be strictly treated for all parties involved. So delivering nothing is always better than delivering a copy.

  • You should develop your model using Keras in the Tensorflow library. So please, DONOT use "from keras import ...", instead USE "from tensorflow.keras import ...".
  • Both public and private test sets are of shape (n_samples, 32, 32, 3) representing RGB images with values in the range [0, 255], same as the training set.
  • The submitted model will be used as follows "model.predict(x)", while x is of shape (n_samples, 32, 32, 3).
  • You can load the data using the NumPy function: "xs = np.load('xs.npy')".
  • It is not allowed to use any other dataset except the provided ones. Using an external dataset will result in disqualification from the competition.
  • Failed submissions aren't counted.

To Submit:

  • After developing and training your model, save it using the function: "model.save('model.h5')", Take care, it should have the name 'model.h5'.
  • Zip the model 'model.h5' and the code 'train.py' to get 'Archive.zip'. DONOT put them in a folder then zip the folder, just zip the h5 file and the script file as they are. Please take care that the names should be the same. The code 'train.py' should produce 'model.h5' after running it.
  • Make sure that your code is reproducible, as we will check it at the end. Check this link.
  • We are aware that running the code on GPU will produce some randomness in the results but it is with a small variability and negligible.
  • Submit the file 'Archive.zip' and wait for the response.

We will list several topics and a few references associated with it. You are free to use these references or any others as long as you won't violate the competition rules.

 

1- Unsupervised deep learning:

2- Transfer Learning and Domain Adaptation

Phase 1. Development

Start: Nov. 15, 2020, midnight

Phase 2. Evaluation

Start: Dec. 14, 2020, midnight

Competition Ends

Dec. 14, 2020, 11 p.m.

You must be logged in to participate in competitions.

Sign In
# Username Score
1 dfokin 0.7813
2 YaroslavPlaskin 0.7490
3 Mostafa_Hegazy 0.7271