The 5th Whole Brain Architecture Hackathon
First phase
ModelEnd
Competition Ends5th WBA Hackathon: Working Memory
The challenge in the 5th Hackathon is to develop biologically inspired models of Working Memory and demonstrate their capabilities in a set of psychology tests routinely used on humans and animals. A variety of Delayed Match to Sample (DMTS) tasks will be selected for this challenge.
In these tasks, the software agent must choose to remember specific features on a screen, maintain these memories for a period of time, use them to complete the task, and then discard them when appropriate. These capabilities must be learned.
In addition, there are two extra elements of difficulty.
- The rules of the game will change over time. The agent must learn to recognize these rule changes and adapt gameplay strategy, and WM behaviour, accordingly.
- The agent has an active vision system that only sees part of the screen at any time. The agent must learn strategies to control & schedule its gaze to obtain the necessary information. It also implies that WM must be used to selectively retain sequences of input.
You must register from the CFP Page: https://wba-initiative.org/en/18626/
Technical information is found on the Hackathon Wiki: https://github.com/wbap/WM_Hackathon/wiki
Evaluation
The competition submission allows you to submit your code alongside a pre-trained agent model checkpoint.
Preparing Submission
To prepare for submitting your code to CodaLab, ensure that you've met all the naming requirements to ensure that your model is properly evaluated. See below:
Naming Restrictions
- The class name `Agent` and filepath `agent/agent.py` class must remain the same
- The class name `AgentEnv` and filepath `gym_game/envs/agent_env.py` must remain the same
- The `model_name` variable passed must be called `agent_model`
- The `preprocessor_name` variable (if used) must be called `obs_preprocessor`
- The following configuration filenames should not be changed:
- `agent_av_pretrained.json`: ensure that you specify your valid checkpoint path here, with the correct configuration
- `agent_env_av.json`
- `dm2s_env_*.json`: this includes `color`, `position` and `shape` variants
- `m2s_env_*.json`: this includes `color`, `position` and `shape` variants
It is advisable to remove any unnecessary logs, files or other data that was produced during training, in order to reduce the overall size of your submission. For example, TensorBoard events generated during training can be significantly large in size, or additional intermediate model checkpoints.
After cleaning up the submission folder, you can compress the contents into a ZIP file. Ensure that you are archiving the actual contents of the directory, rather than the directory itself.
Head over to the 'Particpate' section in the CodaLab competition, and then to 'Submit / View Results'. Clicking the 'Submit' button will prompt you to select the submission archive ZIP file. Depending on the size, it may take a few minutes to be submitted and CodaLab does not provide a user friendly visual indication of this.
The submissions are evaluated by using the pre-trained agent model and evaluating its performance on a number of episodes, and the final score will be the mean episode reward.
Evaluation Details
The evaluation script will import your custom agent model, by importing the `Agent` and `AgentEnv` classes, and restoring the checkpoint specified in `agent_av_pretrained.json`. It will run multiple evaluation runs, one for each mode (color, position, shape) within each game type (Delayed Match to Sample and Match to Sample). The individual run score will be averaged over the 10 trials, and total score will be computed as well.
Competition Rules
Please follow the instruction at the CFP page.
Model
Start: Feb. 15, 2020, midnight
Description: Submit your codebase, including your pre-trained agent checkpoint. Please remove any unnecessary files/logs (e.g. TensorBoard) to reduce file size.
Competition Ends
Nov. 1, 2021, 3 p.m.
You must be logged in to participate in competitions.
Sign In