Benchmarking

This guide explains how to write and run garage benchmark scripts.

Prerequisites

Firstly, you should have installed garage. If not, you can see instructions in the latest documentation to install garage.

Install garage-benchmarks

cd garage/benchmarks && pip install -e .

Write garage benchmark scripts

Define an experiment function

Use @wrap_experiment to define your experiment. ctxt is used for @wrap_experiment. The second and third parameters of the function should be env_id and seed. You should give your function a good name because it is used as the label name when it comes to plotting. See more examples. You can see detailed documentations of defining experiments here.

@wrap_experiment
def trpo_garage_pytorch(ctxt, env_id, seed):
    ...

Define an execution function using @benchmark and iterate_experiments()

Plotting is enabled by default. You can explicitly disable it by defining @benchmark(plot=False).

@benchmark
def your_benchmarks():
    iterate_experiments(trpo_garage_pytorch, MuJoCo1M_ENV_SET)
    iterate_experiments(trpo_garage_tf, MuJoCo1M_ENV_SET)

The execution function should be placed in one of the following files:

garage/benchmarks/src/garage_benchmarks
                      ├── benchmark_algos.py
                      ├── benchmark_auto.py
                      ├── benchmark_baselines.py
                      ├── benchmark_policies.py
                      └── benchmark_q_functions.py

Run garage benchmarks

To list all the benchmarks, do:

garage_benchmark list

To run a benchmark, do:

garage_benchmark run your_benchmark

For the above example, the results will be saved in ./data/local/benchmarks/your_benchmark/.

Environment sets

Algorithm	Observation	Action	Environment Set
On-policy	Pixel	Discrete	*PIXEL_ENV_SET
Off-policy	Pixel	Discrete	Atari1M
Meta-RL	Non-Pixel	Discrete	*ML_ENV_SET
Multi-Task RL	Non-Pixel	Discrete	*MT_ENV_SET
ALL	Non-Pixel	Discrete	*NON_PIXEL_ENV_SET
ALL	Non-Pixel	Continuous	MuJoCo-1M

PIXEL_ENV_SET = [
 'MemorizeDigits-v0', 'CubeCrash-v0', ^'Acrobot-v1', ^'MountainCar-v0',
 ^'CartPole-v1', ^'LunarLander-v2']

^ Using the wrappers PixelObservationWrapper and FrameStack (n=4)

ML_ENV_SET = [
 'ML1-push-v1', 'ML1-reach-v1', 'ML1-pick-place-v1', 'ML10', 'ML45']

MT_ENV_SET = [
 'ML1-push-v1', 'ML1-reach-v1', 'ML1-pick-place-v1', 'MT10', 'MT50']

See Meta-World for ML1/ML10/ML45/MT10/MT50.

STATE_ENV_SET = [
 'LunarLander-v2', 'CartPole-v1', 'Assault-ramDeterministic-v4',
 'Breakout-ramDeterministic-v4', 'ChopperCommand-ramDeterministic-v4',
 'Tutankham-ramDeterministic-v4']

---

This page was authored by Ruofu Wang (@yeukfu).