Sampling¶
Trainer
gets episodes through Sampling to train the policy. In
Garage, Trainer
uses Sampler
to perform sampling. Sampler
manages Worker
s
and assign specific tasks to them, which is doing rollouts with agents and
environments. You can also implement your own sampler and worker. The
followings introduce the existing samplers and workers in Garage.
Sampler¶
Sampler is responsible for assign sampling jobs to workers. Garage now has
two types of Sampler
s:
LocalSampler
, the default sampler, which runs workers in the main process in serial style. With this sampler, all the sampling tasks will run in the same thread.RaySampler
, the sampler using Ray framework to run distributed workers in parallel style.RaySampler
can not only run workers in different CPUs, but also in different machines across network.
Worker¶
Worker is the basic unit to perform a rollout per step. In paralleling
samplers, each worker will typically run in one exclusive CPU. For most
algorithms, Garage provides two kinds of workers, DefaultWorker
and
VecWorker
. A few algorithms (RL2 and PEARL) use custom workers specific to
that algorithm.
DefaultWorker
, the default worker. It works with one single agent/policy and one single environment in one step.VecWorker
, the worker with Vectorization, which runs multiple instances of the simulation on a single CPU.VecWorker
can compute a batch of actions from a policy regarding multiple environments to reduce of overhead of sampling (e.g. feeding forward a neural network).
Setup Sampler and Worker for a Trainer¶
Setup the sampler and worker for a Trainer
is easy. Just passing
sampler_cls
and worker_class
to trainer.setup()
. The number of workers in
the sampler can be set by the parameter n_workers
.
For VecWorker
, you can set the level of vectorization (i.e. the number of
environments simulated in one step) by setting n_envs
in worker_args
.
from garage.sampler import RaySampler, VecWorker
...
trainer.setup(
algo=algo,
env=env,
sampler_cls=RaySampler,
n_workers=4,
worker_class=VecWorker,
worker_args=dict(n_envs=12)
)
...
In the above example, we choose RaySampler
and VecWorker
, set the number
of workers to 4, and set the level of vectorization to 12. With this
configuration, sampling will run in 4 CPUs in parallel and each worker will
sample 12 actions in one step.
This page was authored by Ruofu Wang (@yeukfu).