Submit in 5 Minutes¶

All aboard! 🚂¶

In the next five minutes, you will submit your first agent to the NeurIPS 2020 Flatland Challenge and will see your name on the leaderboard.

leaderboard

Setup¶

Let’s start by cloning the starter kit:

git clone git@github.com:AIcrowd/flatland-challenge-starter-kit.git
cd flatland-challenge-starter-kit

The starter kit comes with a sample agent which performs random actions. We will see how it works in more details in the next section.

Let’s create a conda environment for Flatland. Install conda first if it is not setup on your machine.

conda env create -f environment.yml # creates the flatland-rl environment
conda activate flatland-rl # activates it

Test your local setup¶

We will now run the agent locally to check that everything works as expected.

Let’s download a few test environments to test the agent. Head to the challenge resources and download the provided test set.

Untar them in ./scratch/test-envs. Your directory structure should be as follow:

./scratch
└── test-envs
    ├── Test_0
    │   ├── Level_0.pkl
    │   └── Level_1.pkl
    ├── Test_1
    │   ├── Level_0.pkl
    │   └── Level_1.pkl
    ├── Test_2
    │   ├── Level_0.pkl
    │   └── Level_1.pkl
    ├── Test_3
    │   ├── Level_0.pkl
    │   └── Level_1.pkl
    ├── ...

We will now replicate the setup used on AIcrowd locally to ensure that your submission will be evaluated without problem. This involves three components: your agent, the evaluator service, and a Redis server to let them communicate.

Redis¶

The communication between your agent and the evaluator is done through a Redis server. You should ensure that a redis server is running unprotected on localhost. Follow these instructions to set it up.

You can check that things are fine by running:

redis-cli ping
> PONG

Evaluator service¶

Let’s start the evaluator service. You should use different terminals for the evaluator and the agent as they will need to run at the same time.

flatland-evaluator --tests ./scratch/test-envs/

The agent¶

You can now start the agent:

# on Linux
export AICROWD_TESTS_FOLDER=<path_to_your_tests_directory>

# or on Windows :
#  SET AICROWD_TESTS_FOLDER=<path_to_your_tests_directory>

python run.py

That’s it! the agent should now start interacting with the evaluator, and you should see the results coming in.

Why so complicated?

This client/server architecture is used to fully isolate your solution from the evaluation infrastructure.

Submitting!¶

Let’s first consider the big picture. When taking part in this challenge, you will first train an agent on your local machine, then submit it for evaluation.

The submission process uses git. The code for your agent will be hosted on gitlab.aicrowd.com. When you want your agent to be evaluated, you will push a tag to this repository, which will trigger an evaluation. You are allowed to submit up to 5 submissions per day.

Create your repository¶

Head to https://gitlab.aicrowd.com/ to create your private repository. You can use any name you want for it.

Add your SSH Keys to your GitLab account by following these instructions.

If you do not have SSH Keys, you will first need to generate one.

Push a tag¶

You can then create a submission by making a tag push to your repository on gitlab.aicrowd.com.

Any tag push (where the tag name begins with “submission-“) to your private repository is considered as a submission

First add the a git remote pointing to your repository on gitlab.aicrowd.com

cd flatland-challenge-starter-kit
# Add AIcrowd git remote endpoint
git remote add aicrowd git@gitlab.aicrowd.com:<YOUR_AICROWD_USER_NAME>/flatland-challenge-starter-kit.git
git push aicrowd master

Finally submit by doing:

# Create a tag for your submission and push
git tag -am "submission-v0.1" submission-v0.1
git push aicrowd master
git push aicrowd submission-v0.1

Note that if the contents of your repository (latest commit hash) does not change, then pushing a new tag will not trigger a new evaluation.

You now should be able to see the details of your submission at:

https://gitlab.aicrowd.com/<YOUR_AICROWD_USER_NAME>/flatland-challenge-starter-kit/issues

Remember to update your username in the link above! 😉

In the link above, you should start seeing something like this take shape. The whole evaluation can take a bit of time, you will have to be patient! 🕙