Competition

The Algonauts Project 2021 Challenge - Mini Track (9 ROIs)

Organized by alexlascelles - Current server time: March 30, 2025, 9:24 a.m. UTC

Current

Challenge Phase
May 1, 2021, 3:59 a.m. UTC

End

Competition Ends
Jan. 1, 2030, 3:59 a.m. UTC

<< UPDATE: Although the Algonauts Challenge 2021 Mini Track officially ended on August 15th, we have re-enabled submissions to allow you to test and improve your models further! You can find the final leaderboard for the challenge here. >>

----

This is the submission page for the MINI Track (9 ROIs) of the Algonauts Project 2021 Challenge: How the Human Brain Makes Sense of a World in Motion. More information for the Challenge and Workshop can be found on our website here.

Challenge Overview

The goal of the Mini Track is to predict human brain responses to a set of 3-second videos in 9 regions of the visual brain using computational models.

 

Data and Development Kit

Training Set
1,000 3-second videos + fMRI human brain data of 10 subjects in response to viewing muted videos from this set. The data is provided for 9 regions of the visual brain (V1, V2, V3, V4, LOC, EBA, FFA, STS, PPA) in a Pickle file (e.g. V1.pkl) that contains a num_videos x num_repetitions x num_voxels matrix. For each ROI, we selected voxels that showed significant split-half reliability.
Test Set
102 3-second videos only (.mp4 format).
Development Kit
We provide a development kit to obtain baseline performance provided in the leaderboard. It consists of Python scripts to:
- Extract activations of a computational model (AlexNet) in response to video clips.
- Train a voxel-wise encoding model using AlexNet activations to predict brain responses, and to evaluate the encoding model using cross-validation on the training data.
- Prepare the predicted brain responses from the trained encoding model in the format required for submission to the challenge.
 
The fMRI brain responses of the test set are withheld. 

We describe below the use of training data for model submission to the challenge.
 
 

Predicting Brain Responses: An Example Approach

In the 2021 version we leave it up to you to determine the approach to predict brain responses (see Challenge Rules). However, here we provide an implementation of a common approach called a voxel-wise encoding model (Naselaris et al., 2011; Wu et al., 2006) where the response of each voxel is predicted independently using the multiple features provided by a computational model (a regularized linear regression is typically used to form the prediction).
 
We provide an example implementation of the voxel-wise encoding model using AlexNet as the computational model in the development kit.
 
 

Submission of Predicted Brain Responses

A participant's submission is evaluated against the held-out test set brain responses. For a single submission, upload one .pkl file that comprises predicted brain responses for all 9 ROIs and all 10 subjects. The submission format is provided here. Please follow the Challenge Rules during the submission process.

To participate, click here to submit your predictions.

 
 

Evaluation of the Prediction

To determine how well your models can predict brain responses we compare your submitted synthetic brain data (i.e., those predicted from your model to the left out video clips) to the empirically measured brain responses. The comparison is carried out using Pearson’s correlation, comparing for each voxel the 102-dimensional vector formed by the activations for the 102 test set video clips.  We normalize the correlation value by the square root of corresponding voxel’s split-half reliability. This results in one value of normalized correlation per voxel ranging from -1 to 1.
 
We average the noise normalized correlation across all the voxels in each of the 9 brain regions of interest. Aggregated across 9 visual brain regions, this results in a challenge score that determines the relative place in the leaderboard. View results and the current leaderboard.

We recommend that you start by reviewing the development kit, which includes an example of model comparison to brain responses.
 

Important Dates:

  • Challenge Submission Deadline: August 15, 2021 at 11:59pm (UTC-4)
  • Report Submission Deadline: August 22, 2021 - Submission Link
  • Challenge results released: August 23, 2021

Model Evaluation

For each voxel in an ROI, we calculate the noise normalized correlation, by computing Pearson’s correlation between the predicted voxel’s responses from your model to the left out video clips) and the empirically measured voxel’s response followed by normalization by the square root of split-half reliability. To estimate split-half reliability, the voxel responses to test videos are split into all possible combinations of two splits of 5 repetitions each and the Pearson correlation (ρ) between the splits is calculated. The split-half reliability is then calculated using Spearman-Brown formula (2ρ/(1+ρ)). Reliability of the voxel is the average reliability obtained across all combinations of splits. We then aggregate noise normalized correlation value across voxels in an ROI into one average value, and then average values across ROIs to determine the score of the leaderboard.

Challenge Rules

1. Participants can use any (external) data for model building, any model, and any procedure to predict brain data with one exception: Participants that use the test set for training will be disqualified (in particular brain data generated using the test set).

2. Each participant (single researchers or team, there is no maximum team size) can make 1 submission per day per track.

3. Participants must upload a short report (~4-8 pages) describing their model building process for their best model to a preprint server (e.g. bioRxiv, arXiv), or send a PDF by email to algonauts.mit@gmail.com.
Please use this form to submit the challenge report within 7 days of the challenge submission deadline to be considered for the evaluation of challenge outcomes. Participants that do not make their approach open and transparent cannot be considered. We additionally encourage participants to make their code available online (e.g. on Github) and link to this during their form submission.

We recommend you start by reviewing the provided development kit, which includes Python scripts to extract activations, train an encoding model, and prepare the predicted brain activity from the trained model in the format required for submission to the challenge.

Model Submission

To upload the participant's submission file for the Mini Track (9 ROIs), please follow the following steps;

Step 1

Participants compute predicted fMRI activity from their model for all reliable voxels in all 9 ROIs and for all 10 subjects. The reliable voxels are the ones in the brain data for the training set provided in the Development Kit.

Please prepare a single pickle (.pkl) file named "mini_track.pkl" as demonstrated in the Development Kit. The pickle file is a dictionary of dictionaries containing the predicted fMRI data for all ROIs and all subjects. For example, if we load the pickle file using results = pickle.load(f), then results['V1']['sub10'] should be a numpy array containing the predicted data of all reliable voxels in ROI V1 and subject 10.

Step 2

Please compress "mini_track.pkl" in a single .zip file.

Step 3

On the challenge page, click Participate tab and then click Submit/View Results. It opens a window for you to browse to the location of the stored .zip file and upload it.

Challenge Phase

Start: May 1, 2021, 3:59 a.m.

Description: Each brain region is scored based on their noise-normalized correlation with our held-out brain data. The challenge score is the average noise-normalized correlation across all 9 brain regions. The entry with the highest challenge score is ranked first.

Competition Ends

Jan. 1, 2030, 3:59 a.m.

You must be logged in to participate in competitions.

Sign In