Welcome to Spinning Up in Deep RL!Â¶
IntroductionÂ¶
Table of Contents
What This IsÂ¶
Welcome to Spinning Up in Deep RL! This is an educational resource produced by OpenAI that makes it easier to learn about deep reinforcement learning (deep RL).
For the unfamiliar: reinforcement learning (RL) is a machine learning approach for teaching agents how to solve tasks by trial and error. Deep RL refers to the combination of RL with deep learning.
This module contains a variety of helpful resources, including:
 a short introduction to RL terminology, kinds of algorithms, and basic theory,
 an essay about how to grow into an RL research role,
 a curated list of important papers organized by topic,
 a welldocumented code repo of short, standalone implementations of key algorithms,
 and a few exercises to serve as warmups.
Why We Built ThisÂ¶
One of the single most common questions that we hear is
If I want to contribute to AI safety, how do I get started?
At OpenAI, we believe that deep learning generally—and deep reinforcement learning specifically—will play central roles in the development of powerful AI technology. To ensure that AI is safe, we have to come up with safety strategies and algorithms that are compatible with this paradigm. As a result, we encourage everyone who asks this question to study these fields.
However, while there are many resources to help people quickly ramp up on deep learning, deep reinforcement learning is more challenging to break into. To begin with, a student of deep RL needs to have some background in math, coding, and regular deep learning. Beyond that, they need both a highlevel view of the field—an awareness of what topics are studied in it, why they matter, and what’s been done already—and careful instruction on how to connect algorithm theory to algorithm code.
The highlevel view is hard to come by because of how new the field is. There is not yet a standard deep RL textbook, so most of the knowledge is locked up in either papers or lecture series, which can take a long time to parse and digest. And learning to implement deep RL algorithms is typically painful, because either
 the paper that publishes an algorithm omits or inadvertently obscures key design details,
 or widelypublic implementations of an algorithm are hard to read, hiding how the code lines up with the algorithm.
While fantastic repos like garage, Baselines, and rllib make it easier for researchers who are already in the field to make progress, they build algorithms into frameworks in ways that involve many nonobvious choices and tradeoffs, which makes them hard to learn from. Consequently, the field of deep RL has a pretty high barrier to entry—for new researchers as well as practitioners and hobbyists.
So our package here is designed to serve as the missing middle step for people who are excited by deep RL, and would like to learn how to use it or make a contribution, but don’t have a clear sense of what to study or how to transmute algorithms into code. We’ve tried to make this as helpful a launching point as possible.
That said, practitioners aren’t the only people who can (or should) benefit from these materials. Solving AI safety will require people with a wide range of expertise and perspectives, and many relevant professions have no connection to engineering or computer science at all. Nonetheless, everyone involved will need to learn enough about the technology to make informed decisions, and several pieces of Spinning Up address that need.
How This Serves Our MissionÂ¶
OpenAI’s mission is to ensure the safe development of AGI and the broad distribution of benefits from AI more generally. Teaching tools like Spinning Up help us make progress on both of these objectives.
To begin with, we move closer to broad distribution of benefits any time we help people understand what AI is and how it works. This empowers people to think critically about the many issues we anticipate will arise as AI becomes more sophisticated and important in our lives.
Also, critically, we need people to help us work on making sure that AGI is safe. This requires a skill set which is currently in short supply because of how new the field is. We know that many people are interested in helping us, but don’t know how—here is what you should study! If you can become an expert on this material, you can make a difference on AI safety.
Code Design PhilosophyÂ¶
The algorithm implementations in the Spinning Up repo are designed to be
 as simple as possible while still being reasonably good,
 and highlyconsistent with each other to expose fundamental similarities between algorithms.
They are almost completely selfcontained, with virtually no common code shared between them (except for logging, saving, loading, and MPI utilities), so that an interested person can study each algorithm separately without having to dig through an endless chain of dependencies to see how something is done. The implementations are patterned so that they come as close to pseudocode as possible, to minimize the gap between theory and code.
Importantly, they’re all structured similarly, so if you clearly understand one, jumping into the next is painless.
We tried to minimize the number of tricks used in each algorithm’s implementation, and minimize the differences between otherwisesimilar algorithms. To give some examples of removed tricks: we omit regularization terms present in the original SoftActor Critic code, as well as observation normalization from all algorithms. For an example of where we’ve removed differences between algorithms: our implementations of DDPG, TD3, and SAC all follow a convention of running gradient descent updates after fixed intervals of environment interaction. (By contrast, other public implementations of these algorithms usually take slightly different approaches from each other, making them a little bit harder to compare.)
All algorithms are “reasonably good” in the sense that they achieve roughly the intended performance, but don’t necessarily match the best reported results in the literature on every task. Consequently, be careful if using any of these implementations for scientific benchmarking comparisons. Details on each implementation’s specific performance level can be found on our benchmarks page.
LongTerm Support and Support HistoryÂ¶
Spinning Up is currently in maintenance mode. If there are any breaking bugs, we’ll repair them to ensure that Spinning Up can continue to help people study deep RL.
Support history so far:
Nov 8, 2018: Initial release!
Nov, 2018: Release was followed by a threeweek period of highbandwidth support.
April, 2019: Approximately six months after release, we conducted an internal review of Spinning Up based on feedback from the community. The review surfaced interest in a few key features:
 Implementations in Other Neural Network Libraries. Several people expressed interest in seeing Spinning Up use alternatives to Tensorflow v1 for the RL implementations. A few members of the community even developed their own PyTorch versions of Spinning Up algorithms, such as Kashif Rasul’s Fired Up, Kai Arulkumaran’s Spinning Up Basic, and Misha Laskin’s Torching Up. As a result, making this kind of “Rosetta Stone” for deep RL became a high priority for future work.
 Open Source RL Environments. Many people expressed an interest in seeing Spinning Up use more open source RL environments (eg PyBullet) for benchmarks, examples, and exercises.
 More Algorithms. There was some interest in seeing other algorithms included in Spinning Up, especially Deep QNetworks.
Jan, 2020: The PyTorch update to Spinning Up was released!
Future: No major updates are currently planned for Spinning Up. In the event it makes sense for us to release an additional update, following what we found in the 6month review, the nexthighest priority features are to focus more on open source RL environments and adding algorithms.
Additionally, as discussed in the blog post, Spinning Up has been integrated into the curriculum for our Scholars and Fellows programs.
InstallationÂ¶
Table of Contents
Spinning Up requires Python3, OpenAI Gym, and OpenMPI.
Spinning Up is currently only supported on Linux and OSX. It may be possible to install on Windows, though this hasn’t been extensively tested. [1]
You Should Know
Many examples and benchmarks in Spinning Up refer to RL environments that use the MuJoCo physics engine. MuJoCo is a proprietary software that requires a license, which is free to trial and free for students, but otherwise is not free. As a result, installing it is optional, but because of its importance to the research community—it is the de facto standard for benchmarking deep RL algorithms in continuous control—it is preferred.
Don’t worry if you decide not to install MuJoCo, though. You can definitely get started in RL by running RL algorithms on the Classic Control and Box2d environments in Gym, which are totally free to use.
[1]  It looks like at least one person has figured out a workaround for running on Windows. If you try another way and succeed, please let us know how you did it! 
Installing PythonÂ¶
We recommend installing Python through Anaconda. Anaconda is a library that includes Python and many useful packages for Python, as well as an environment manager called conda that makes package management simple.
Follow the installation instructions for Anaconda here. Download and install Anaconda3 (at time of writing, Anaconda35.3.0). Then create a conda Python 3.6 env for organizing packages used in Spinning Up:
conda create n spinningup python=3.6
To use Python from the environment you just created, activate the environment with:
conda activate spinningup
You Should Know
If you’re new to python environments and package management, this stuff can quickly get confusing or overwhelming, and you’ll probably hit some snags along the way. (Especially, you should expect problems like, “I just installed this thing, but it says it’s not found when I try to use it!”) You may want to read through some clean explanations about what package management is, why it’s a good idea, and what commands you’ll typically have to execute to correctly use it.
FreeCodeCamp has a good explanation worth reading. There’s a shorter description on Towards Data Science which is also helpful and informative. Finally, if you’re an extremely patient person, you may want to read the (dry, but very informative) documentation page from Conda.
Installing Spinning UpÂ¶
git clone https://github.com/openai/spinningup.git
cd spinningup
pip install e .
You Should Know
Spinning Up defaults to installing everything in Gym except the MuJoCo environments. In case you run into any trouble with the Gym installation, check out the Gym github page for help. If you want the MuJoCo environments, see the optional installation section below.
Check Your InstallÂ¶
To see if you’ve successfully installed Spinning Up, try running PPO in the LunarLanderv2 environment with
python m spinup.run ppo hid "[32,32]" env LunarLanderv2 exp_name installtest gamma 0.999
This might run for around 10 minutes, and you can leave it going in the background while you continue reading through documentation. This won’t train the agent to completion, but will run it for long enough that you can see some learning progress when the results come in.
After it finishes training, watch a video of the trained policy with
python m spinup.run test_policy data/installtest/installtest_s0
And plot the results with
python m spinup.run plot data/installtest/installtest_s0
Installing MuJoCo (Optional)Â¶
First, go to the mujocopy github page. Follow the installation instructions in the README, which describe how to install the MuJoCo physics engine and the mujocopy package (which allows the use of MuJoCo from Python).
You Should Know
In order to use the MuJoCo simulator, you will need to get a MuJoCo license. Free 30day licenses are available to anyone, and free 1year licenses are available to fulltime students.
Once you have installed MuJoCo, install the corresponding Gym environments with
pip install gym[mujoco,robotics]
And then check that things are working by running PPO in the Walker2dv2 environment with
python m spinup.run ppo hid "[32,32]" env Walker2dv2 exp_name mujocotest
AlgorithmsÂ¶
Table of Contents
What’s IncludedÂ¶
The following algorithms are implemented in the Spinning Up package:
 Vanilla Policy Gradient (VPG)
 Trust Region Policy Optimization (TRPO)
 Proximal Policy Optimization (PPO)
 Deep Deterministic Policy Gradient (DDPG)
 Twin Delayed DDPG (TD3)
 Soft ActorCritic (SAC)
They are all implemented with MLP (nonrecurrent) actorcritics, making them suitable for fullyobserved, nonimagebased RL environments, e.g. the Gym Mujoco environments.
Spinning Up has two implementations for each algorithm (except for TRPO): one that uses PyTorch as the neural network library, and one that uses Tensorflow v1 as the neural network library. (TRPO is currently only available in Tensorflow.)
Why These Algorithms?Â¶
We chose the core deep RL algorithms in this package to reflect useful progressions of ideas from the recent history of the field, culminating in two algorithms in particular—PPO and SAC—which are close to state of the art on reliability and sample efficiency among policylearning algorithms. They also expose some of the tradeoffs that get made in designing and using algorithms in deep RL.
The OnPolicy AlgorithmsÂ¶
Vanilla Policy Gradient is the most basic, entrylevel algorithm in the deep RL space because it completely predates the advent of deep RL altogether. The core elements of VPG go all the way back to the late 80s / early 90s. It started a trail of research which ultimately led to stronger algorithms such as TRPO and then PPO soon after.
A key feature of this line of work is that all of these algorithms are onpolicy: that is, they don’t use old data, which makes them weaker on sample efficiency. But this is for a good reason: these algorithms directly optimize the objective you care about—policy performance—and it works out mathematically that you need onpolicy data to calculate the updates. So, this family of algorithms trades off sample efficiency in favor of stability—but you can see the progression of techniques (from VPG to TRPO to PPO) working to make up the deficit on sample efficiency.
The OffPolicy AlgorithmsÂ¶
DDPG is a similarly foundational algorithm to VPG, although much younger—the theory of deterministic policy gradients, which led to DDPG, wasn’t published until 2014. DDPG is closely connected to Qlearning algorithms, and it concurrently learns a Qfunction and a policy which are updated to improve each other.
Algorithms like DDPG and QLearning are offpolicy, so they are able to reuse old data very efficiently. They gain this benefit by exploiting Bellman’s equations for optimality, which a Qfunction can be trained to satisfy using any environment interaction data (as long as there’s enough experience from the highreward areas in the environment).
But problematically, there are no guarantees that doing a good job of satisfying Bellman’s equations leads to having great policy performance. Empirically one can get great performance—and when it happens, the sample efficiency is wonderful—but the absence of guarantees makes algorithms in this class potentially brittle and unstable. TD3 and SAC are descendants of DDPG which make use of a variety of insights to mitigate these issues.
Code FormatÂ¶
All implementations in Spinning Up adhere to a standard template. They are split into two files: an algorithm file, which contains the core logic of the algorithm, and a core file, which contains various utilities needed to run the algorithm.
The algorithm file always starts with a class definition for an experience buffer object, which is used to store information from agentenvironment interactions. Next, there is a single function which runs the algorithm. The algorithm function follows a template that is roughly the same across the PyTorch and Tensorflow versions, but we’ll break it down for each separately below. Finally, there’s some support in each algorithm file for directly running the algorithm in Gym environments from the command line (though this is not the recommended way to run the algorithms—we’ll describe how to do that on the Running Experiments page).
The Algorithm Function: PyTorch VersionÂ¶
The algorithm function for a PyTorch implementation performs the following tasks in (roughly) this order:
 Logger setup
 Random seed setting
 Environment instantiation
 Constructing the actorcritic PyTorch module via the
actor_critic
function passed to the algorithm function as an argument Instantiating the experience buffer
 Setting up callable loss functions that also provide diagnostics specific to the algorithm
 Making PyTorch optimizers
 Setting up model saving through the logger
 Setting up an update function that runs one epoch of optimization or one step of descent
 Running the main loop of the algorithm:
 Run the agent in the environment
 Periodically update the parameters of the agent according to the main equations of the algorithm
 Log key performance metrics and save agent
The Algorithm Function: Tensorflow VersionÂ¶
The algorithm function for a Tensorflow implementation performs the following tasks in (roughly) this order:
 Logger setup
 Random seed setting
 Environment instantiation
 Making placeholders for the computation graph
 Building the actorcritic computation graph via the
actor_critic
function passed to the algorithm function as an argument Instantiating the experience buffer
 Building the computation graph for loss functions and diagnostics specific to the algorithm
 Making training ops
 Making the TF Session and initializing parameters
 Setting up model saving through the logger
 Defining functions needed for running the main loop of the algorithm (e.g. the core update function, get action function, and test agent function, depending on the algorithm)
 Running the main loop of the algorithm:
 Run the agent in the environment
 Periodically update the parameters of the agent according to the main equations of the algorithm
 Log key performance metrics and save agent
The Core FileÂ¶
The core files don’t adhere as closely as the algorithms files to a template, but do have some approximate structure:
 Tensorflow only: Functions related to making and managing placeholders
 Functions for building sections of computation graph relevant to the
actor_critic
method for a particular algorithm Any other useful functions
 Implementations for an MLP actorcritic compatible with the algorithm, where both the policy and the value function(s) are represented by simple MLPs
Running ExperimentsÂ¶
Table of Contents
One of the best ways to get a feel for deep RL is to run the algorithms and see how they perform on different tasks. The Spinning Up code library makes smallscale (local) experiments easy to do, and in this section, we’ll discuss two ways to run them: either from the command line, or through function calls in scripts.
Launching from the Command LineÂ¶
Spinning Up ships with spinup/run.py
, a convenient tool that lets you easily launch any algorithm (with any choices of hyperparameters) from the command line. It also serves as a thin wrapper over the utilities for watching trained policies and plotting, although we will not discuss that functionality on this page (for those details, see the pages on experiment outputs and plotting).
The standard way to run a Spinning Up algorithm from the command line is
python m spinup.run [algo name] [experiment flags]
eg:
python m spinup.run ppo env Walker2dv2 exp_name walker
You Should Know
If you are using ZShell: ZShell interprets square brackets as special characters. Spinning Up uses square brackets in a few ways for command line arguments; make sure to escape them, or try the solution recommended here if you want to escape them by default.
Detailed Quickstart Guide
python m spinup.run ppo exp_name ppo_ant env Antv2 clip_ratio 0.1 0.2
hid[h] [32,32] [64,32] act torch.nn.Tanh seed 0 10 20 dt
data_dir path/to/data
runs PPO in the Antv2
Gym environment, with various settings controlled by the flags.
By default, the PyTorch version will run (except for with TRPO, since Spinning Up doesn’t have a PyTorch TRPO yet). Substitute ppo
with ppo_tf1
for the Tensorflow version.
clip_ratio
, hid
, and act
are flags to set some algorithm hyperparameters. You can provide multiple values for hyperparameters to run multiple experiments. Check the docs to see what hyperparameters you can set (click here for the PPO documentation).
hid
and act
are special shortcut flags for setting the hidden sizes and activation function for the neural networks trained by the algorithm.
The seed
flag sets the seed for the random number generator. RL algorithms have high variance, so try multiple seeds to get a feel for how performance varies.
The dt
flag ensures that the save directory names will have timestamps in them (otherwise they don’t, unless you set FORCE_DATESTAMP=True
in spinup/user_config.py
).
The data_dir
flag allows you to set the save folder for results. The default value is set by DEFAULT_DATA_DIR
in spinup/user_config.py
, which will be a subfolder data
in the spinningup
folder (unless you change it).
Save directory names are based on exp_name
and any flags which have multiple values. Instead of the full flag, a shorthand will appear in the directory name. Shorthands can be provided by the user in square brackets after the flag, like hid[h]
; otherwise, shorthands are substrings of the flag (clip_ratio
becomes cli
). To illustrate, the save directory for the run with clip_ratio=0.1
, hid=[32,32]
, and seed=10
will be:
path/to/data/YYMMDD_ppo_ant_cli01_h3232/YYMMDD_HHMMSSppo_ant_cli01_h3232_seed10
Choosing PyTorch or Tensorflow from the Command LineÂ¶
To use a PyTorch version of an algorithm, run with
python m spinup.run [algo]_pytorch
To use a Tensorflow version of an algorithm, run with
python m spinup.run [algo]_tf1
If you run python m spinup.run [algo]
without _pytorch
or _tf1
, the runner will look in spinup/user_config.py
for which version it should default to for that algorithm.
Setting Hyperparameters from the Command LineÂ¶
Every hyperparameter in every algorithm can be controlled directly from the command line. If kwarg
is a valid keyword arg for the function call of an algorithm, you can set values for it with the flag kwarg
. To find out what keyword args are available, see either the docs page for an algorithm, or try
python m spinup.run [algo name] help
to see a readout of the docstring.
You Should Know
Values pass through eval()
before being used, so you can describe some functions and objects directly from the command line. For example:
python m spinup.run ppo env Walker2dv2 exp_name walker act torch.nn.ELU
sets torch.nn.ELU
as the activation function. (Tensorflow equivalent: run ppo_tf1
with act tf.nn.elu
.)
You Should Know
There’s some nice handling for kwargs that take dict values. Instead of having to provide
key dict(v1=value_1, v2=value_2)
you can give
key:v1 value_1 key:v2 value_2
to get the same result.
Launching Multiple Experiments at OnceÂ¶
You can launch multiple experiments, to be executed in series, by simply providing more than one value for a given argument. (An experiment for each possible combination of values will be launched.)
For example, to launch otherwiseequivalent runs with different random seeds (0, 10, and 20), do:
python m spinup.run ppo env Walker2dv2 exp_name walker seed 0 10 20
Experiments don’t launch in parallel because they soak up enough resources that executing several at the same time wouldn’t get a speedup.
Special FlagsÂ¶
A few flags receive special treatment.
Environment FlagÂ¶

env
,
env_name
Â¶
string. The name of an environment in the OpenAI Gym. All Spinning Up algorithms are implemented as functions that accept
env_fn
as an argument, whereenv_fn
must be a callable function that builds a copy of the RL environment. Since the most common use case is Gym environments, though, all of which are built throughgym.make(env_name)
, we allow you to just specifyenv_name
(orenv
for short) at the command line, which gets converted to a lambdafunction that builds the correct gym environment.
Shortcut FlagsÂ¶
Some algorithm arguments are relatively long, and we enabled shortcuts for them:

hid
,
ac_kwargs
:hidden_sizes
Â¶ list of ints. Sets the sizes of the hidden layers in the neural networks (policies and value functions).

act
,
ac_kwargs
:activation
Â¶ tf op. The activation function for the neural networks in the actor and critic.
These flags are valid for all current Spinning Up algorithms.
Config FlagsÂ¶
These flags are not hyperparameters of any algorithm, but change the experimental configuration in some way.

cpu
,
num_cpu
Â¶
int. If this flag is set, the experiment is launched with this many processes, one per cpu, connected by MPI. Some algorithms are amenable to this sort of parallelization but not all. An error will be raised if you try setting
num_cpu
> 1 for an incompatible algorithm. You can also setnum_cpu auto
, which will automatically use as many CPUs as are available on the machine.

exp_name
Â¶
string. The experiment name. This is used in naming the save directory for each experiment. The default is “cmd” + [algo name].

data_dir
Â¶
path. Set the base save directory for this experiment or set of experiments. If none is given, the
DEFAULT_DATA_DIR
inspinup/user_config.py
will be used.

datestamp
Â¶
bool. Include date and time in the name for the save directory of the experiment.
Where Results are SavedÂ¶
Results for a particular experiment (a single run of a configuration of hyperparameters) are stored in
data_dir/[outer_prefix]exp_name[suffix]/[inner_prefix]exp_name[suffix]_s[seed]
where
data_dir
is the value of thedata_dir
flag (defaults toDEFAULT_DATA_DIR
fromspinup/user_config.py
ifdata_dir
is not given), the
outer_prefix
is aYYMMDD_
timestamp if thedatestamp
flag is raised, otherwise nothing,  the
inner_prefix
is aYYMMDD_HHMMSS
timestamp if thedatestamp
flag is raised, otherwise nothing,  and
suffix
is a special string based on the experiment hyperparameters.
How is Suffix Determined?Â¶
Suffixes are only included if you run multiple experiments at once, and they only include references to hyperparameters that differ across experiments, except for random seed. The goal is to make sure that results for similar experiments (ones which share all params except seed) are grouped in the same folder.
Suffixes are constructed by combining shorthands for hyperparameters with their values, where a shorthand is either 1) constructed automatically from the hyperparameter name or 2) supplied by the user. The user can supply a shorthand by writing in square brackets after the kwarg flag.
For example, consider:
python m spinup.run ddpg_tf1 env Hopperv2 hid[h] [300] [128,128] act tf.nn.tanh tf.nn.relu
Here, the hid
flag is given a usersupplied shorthand, h
. The act
flag is not given a shorthand by the user, so one will be constructed for it automatically.
The suffixes produced in this case are:
_h128128_acactrelu
_h128128_acacttanh
_h300_acactrelu
_h300_acacttanh
Note that the h
was given by the user. the acact
shorthand was constructed from ac_kwargs:activation
(the true name for the act
flag).
ExtraÂ¶
You Don’t Actually Need to Know This One
Each individual algorithm is located in a file spinup/algos/BACKEND/ALGO_NAME/ALGO_NAME.py
, and these files can be run directly from the command line with a limited set of arguments (some of which differ from what’s available to spinup/run.py
). The command line support in the individual algorithm files is essentially vestigial, however, and this is not a recommended way to perform experiments.
This documentation page will not describe those command line calls, and will only describe calls through spinup/run.py
.
Launching from ScriptsÂ¶
Each algorithm is implemented as a python function, which can be imported directly from the spinup
package, eg
>>> from spinup import ppo_pytorch as ppo
See the documentation page for each algorithm for a complete account of possible arguments. These methods can be used to set up specialized custom experiments, for example:
from spinup import ppo_tf1 as ppo
import tensorflow as tf
import gym
env_fn = lambda : gym.make('LunarLanderv2')
ac_kwargs = dict(hidden_sizes=[64,64], activation=tf.nn.relu)
logger_kwargs = dict(output_dir='path/to/output_dir', exp_name='experiment_name')
ppo(env_fn=env_fn, ac_kwargs=ac_kwargs, steps_per_epoch=5000, epochs=250, logger_kwargs=logger_kwargs)
Using ExperimentGridÂ¶
It’s often useful in machine learning research to run the same algorithm with many possible hyperparameters. Spinning Up ships with a simple tool for facilitating this, called ExperimentGrid.
Consider the example in spinup/examples/pytorch/bench_ppo_cartpole.py
:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19  from spinup.utils.run_utils import ExperimentGrid
from spinup import ppo_pytorch
import torch
if __name__ == '__main__':
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('cpu', type=int, default=4)
parser.add_argument('num_runs', type=int, default=3)
args = parser.parse_args()
eg = ExperimentGrid(name='ppopytbench')
eg.add('env_name', 'CartPolev0', '', True)
eg.add('seed', [10*i for i in range(args.num_runs)])
eg.add('epochs', 10)
eg.add('steps_per_epoch', 4000)
eg.add('ac_kwargs:hidden_sizes', [(32,), (64,64)], 'hid')
eg.add('ac_kwargs:activation', [torch.nn.Tanh, torch.nn.ReLU], '')
eg.run(ppo_pytorch, num_cpu=args.cpu)

(An equivalent Tensorflow example is available in spinup/examples/tf1/bench_ppo_cartpole.py
.)
After making the ExperimentGrid object, parameters are added to it with
eg.add(param_name, values, shorthand, in_name)
where in_name
forces a parameter to appear in the experiment name, even if it has the same value across all experiments.
After all parameters have been added,
eg.run(thunk, **run_kwargs)
runs all experiments in the grid (one experiment per valid configuration), by providing the configurations as kwargs to the function thunk
. ExperimentGrid.run
uses a function named call_experiment to launch thunk
, and **run_kwargs
specify behaviors for call_experiment
. See the documentation page for details.
Except for the absence of shortcut kwargs (you can’t use hid
for ac_kwargs:hidden_sizes
in ExperimentGrid
), the basic behavior of ExperimentGrid
is the same as running things from the command line. (In fact, spinup.run
uses an ExperimentGrid
under the hood.)
Experiment OutputsÂ¶
Table of Contents
In this section we’ll cover
 what outputs come from Spinning Up algorithm implementations,
 what formats they’re stored in and how they’re organized,
 where they are stored and how you can change that,
 and how to load and run trained policies.
You Should Know
Spinning Up implementations currently have no way to resume training for partiallytrained agents. If you consider this feature important, please let us know—or consider it a hacking project!
Algorithm OutputsÂ¶
Each algorithm is set up to save a training run’s hyperparameter configuration, learning progress, trained agent and value functions, and a copy of the environment if possible (to make it easy to load up the agent and environment simultaneously). The output directory contains the following:
Output Directory Structure  
pyt_save/ 
PyTorch implementations only. A directory containing
everything needed to restore the trained agent and value
functions. (Details for PyTorch saves below.)

tf1_save/ 
Tensorflow implementations only. A directory containing
everything needed to restore the trained agent and value
functions. (Details for Tensorflow saves below.)

config.json 
A dict containing an ascompleteaspossible description
of the args and kwargs you used to launch the training
function. If you passed in something which can’t be
serialized to JSON, it should get handled gracefully by the
logger, and the config file will represent it with a string.
Note: this is meant for recordkeeping only. Launching an
experiment from a config file is not currently supported.

progress.txt 
A tabseparated value file containing records of the metrics
recorded by the logger throughout training. eg,
Epoch ,AverageEpRet , etc. 
vars.pkl 
A pickle file containing anything about the algorithm state
which should get stored. Currently, all algorithms only use
this to save a copy of the environment.

You Should Know
Sometimes environmentsaving fails because the environment can’t be pickled, and vars.pkl
is empty. This is known to be a problem for Gym Box2D environments in older versions of Gym, which can’t be saved in this manner.
You Should Know
As of 1/30/20, the save directory structure has changed slightly. Previously, Tensorflow graphs were saved in the simple_save/
folder; this has been replaced with tf1_save/
.
You Should Know
The only file in here that you should ever have to use “by hand” is the config.json
file. Our agent testing utility will load things from the tf1_save/
or pyt_save/
directory, and our plotter interprets the contents of progress.txt
, and those are the correct tools for interfacing with these outputs. But there is no tooling for config.json
—it’s just there so that if you forget what hyperparameters you ran an experiment with, you can doublecheck.
PyTorch Save Directory InfoÂ¶
The pyt_save
directory contains:
Pyt_Save Directory Structure  
model.pt 
A file created with
torch.save , essentially just apickled PyTorch
nn.Module . Loading it will restorea trained agent as an ActorCritic object with an
act method.

Tensorflow Save Directory InfoÂ¶
The tf1_save
directory contains:
TF1_Save Directory Structure  
variables/ 
A directory containing outputs from the Tensorflow Saver.
See documentation for Tensorflow SavedModel.

model_info.pkl 
A dict containing information (map from key to tensor name)
which helps us unpack the saved model after loading.

saved_model.pb 
A protocol buffer, needed for a Tensorflow SavedModel.

Save Directory LocationÂ¶
Experiment results will, by default, be saved in the same directory as the Spinning Up package, in a folder called data
:
spinningup/ data/ ... docs/ ... spinup/ ... LICENSE setup.py
You can change the default results directory by modifying DEFAULT_DATA_DIR
in spinup/user_config.py
.
Loading and Running Trained PoliciesÂ¶
If Environment Saves SuccessfullyÂ¶
For cases where the environment is successfully saved alongside the agent, it’s a cinch to watch the trained agent act in the environment using:
python m spinup.run test_policy path/to/output_directory
There are a few flags for options:

l
L
,
len
=L
,
default
=0
Â¶ int. Maximum length of test episode / trajectory / rollout. The default of 0 means no maximum episode length—episodes only end when the agent has reached a terminal state in the environment. (Note: setting L=0 will not prevent Gym envs wrapped by TimeLimit wrappers from ending when they reach their preset maximum episode length.)

n
N
,
episodes
=N
,
default
=100
Â¶ int. Number of test episodes to run the agent for.

nr
,
norender
Â¶
Do not render the test episodes to the screen. In this case,
test_policy
will only print the episode returns and lengths. (Use case: the renderer slows down the testing process, and you just want to get a fast sense of how the agent is performing, so you don’t particularly care to watch it.)

i
I
,
itr
=I
,
default
=1
Â¶ int. This is an option for a special case which is not supported by algorithms in this package asshipped, but which they are easily modified to do. Use case: Sometimes it’s nice to watch trained agents from many different points in training (eg watch at iteration 50, 100, 150, etc.). The logger can do this—save snapshots of the agent from those different points, so they can be run and watched later. In this case, you use this flag to specify which iteration to run. But again: spinup algorithms by default only save snapshots of the most recent agent, overwriting the old snapshots.
The default value of this flag means “use the latest snapshot.”
To modify an algo so it does produce multiple snapshots, find the following line (which is present in all of the algorithms):
logger.save_state({'env': env}, None)
and tweak it to
logger.save_state({'env': env}, epoch)
Make sure to then also set
save_freq
to something reasonable (because if it defaults to 1, for instance, you’ll flood your output directory with onesave
folder for each snapshot—which adds up fast).

d
,
deterministic
Â¶
Another special case, which is only used for SAC. The Spinning Up SAC implementation trains a stochastic policy, but is evaluated using the deterministic mean of the action distribution.
test_policy
will default to using the stochastic policy trained by SAC, but you should set the deterministic flag to watch the deterministic mean policy (the correct evaluation policy for SAC). This flag is not used for any other algorithms.
Environment Not Found ErrorÂ¶
If the environment wasn’t saved successfully, you can expect test_policy.py
to crash with something that looks like
Traceback (most recent call last): File "spinup/utils/test_policy.py", line 153, in <module> run_policy(env, get_action, args.len, args.episodes, not(args.norender)) File "spinup/utils/test_policy.py", line 114, in run_policy "and we can't run the agent in it. :( nn Check out the readthedocs " + AssertionError: Environment not found! It looks like the environment wasn't saved, and we can't run the agent in it. :( Check out the readthedocs page on Experiment Outputs for how to handle this situation.
In this case, watching your agent perform is slightly more of a pain but not impossible, as long as you can recreate your environment easily. Try the following in IPython:
>>> from spinup.utils.test_policy import load_policy_and_env, run_policy
>>> import your_env
>>> _, get_action = load_policy_and_env('/path/to/output_directory')
>>> env = your_env.make()
>>> run_policy(env, get_action)
Logging data to /tmp/experiments/1536150702/progress.txt
Episode 0 EpRet 163.830 EpLen 93
Episode 1 EpRet 346.164 EpLen 99
...
Using Trained Value FunctionsÂ¶
The test_policy.py
tool doesn’t help you look at trained value functions, and if you want to use those, you will have to do some digging by hand. For the PyTorch case, load the saved model file with torch.load
and check the documentation for each algorithm to see what modules the ActorCritic object has. For the Tensorflow case, load the saved computation graph with the restore_tf_graph function, and check the documentation for each algorithm to see what functions were saved.
Plotting ResultsÂ¶
Spinning Up ships with a simple plotting utility for interpreting results. Run it with:
python m spinup.run plot [path/to/output_directory ...] [legend [LEGEND ...]]
[xaxis XAXIS] [value [VALUE ...]] [count] [smooth S]
[select [SEL ...]] [exclude [EXC ...]]
Positional Arguments:

logdir
Â¶
strings. As many log directories (or prefixes to log directories, which the plotter will autocomplete internally) as you’d like to plot from. Logdirs will be searched recursively for experiment outputs.
You Should Know
The internal autocompleting is really handy! Suppose you have run several experiments, with the aim of comparing performance between different algorithms, resulting in a log directory structure of:
data/ bench_algo1/ bench_algo1seed0/ bench_algo1seed10/ bench_algo2/ bench_algo2seed0/ bench_algo2seed10/
You can easily produce a graph comparing algo1 and algo2 with:
python spinup/utils/plot.py data/bench_algo
relying on the autocomplete to find both
data/bench_algo1
anddata/bench_algo2
.
Optional Arguments:

l
,
legend
=[LEGEND ...]
Â¶ strings. Optional way to specify legend for the plot. The plotter legend will automatically use the
exp_name
from theconfig.json
file, unless you tell it otherwise through this flag. This only works if you provide a name for each directory that will get plotted. (Note: this may not be the same as the number of logdir args you provide! Recall that the plotter looks for autocompletes of the logdir args: there may be more than one match for a given logdir prefix, and you will need to provide a legend string for each one of those matches—unless you have removed some of them as candidates via selection or exclusion rules (below).)

x
,
xaxis
=XAXIS
,
default
='TotalEnvInteracts'
Â¶ string. Pick what column from data is used for the xaxis.

y
,
value
=[VALUE ...]
,
default
='Performance'
Â¶ strings. Pick what columns from data to graph on the yaxis. Submitting multiple values will produce multiple graphs. Defaults to
Performance
, which is not an actual output of any algorithm. Instead,Performance
refers to eitherAverageEpRet
, the correct performance measure for the onpolicy algorithms, orAverageTestEpRet
, the correct performance measure for the offpolicy algorithms. The plotter will automatically figure out which ofAverageEpRet
orAverageTestEpRet
to report for each separate logdir.

count
Â¶
Optional flag. By default, the plotter shows yvalues which are averaged across all results that share an
exp_name
, which is typically a set of identical experiments that only vary in random seed. But if you’d like to see all of those curves separately, use thecount
flag.

s
,
smooth
=S
,
default
=1
Â¶ int. Smooth data by averaging it over a fixed window. This parameter says how wide the averaging window will be.

select
=[SEL ...]
Â¶ strings. Optional selection rule: the plotter will only show curves from logdirs that contain all of these substrings.

exclude
=[EXC ...]
Â¶ strings. Optional exclusion rule: plotter will only show curves from logdirs that do not contain these substrings.
Part 1: Key Concepts in RLÂ¶
Table of Contents
Welcome to our introduction to reinforcement learning! Here, we aim to acquaint you with
 the language and notation used to discuss the subject,
 a highlevel explanation of what RL algorithms do (although we mostly avoid the question of how they do it),
 and a little bit of the core math that underlies the algorithms.
In a nutshell, RL is the study of agents and how they learn by trial and error. It formalizes the idea that rewarding or punishing an agent for its behavior makes it more likely to repeat or forego that behavior in the future.
What Can RL Do?Â¶
RL methods have recently enjoyed a wide variety of successes. For example, it’s been used to teach computers to control robots in simulation...
...and in the real world...
It’s also famously been used to create breakthrough AIs for sophisticated strategy games, most notably Go and Dota, taught computers to play Atari games from raw pixels, and trained simulated robots to follow human instructions.
Key Concepts and TerminologyÂ¶
The main characters of RL are the agent and the environment. The environment is the world that the agent lives in and interacts with. At every step of interaction, the agent sees a (possibly partial) observation of the state of the world, and then decides on an action to take. The environment changes when the agent acts on it, but may also change on its own.
The agent also perceives a reward signal from the environment, a number that tells it how good or bad the current world state is. The goal of the agent is to maximize its cumulative reward, called return. Reinforcement learning methods are ways that the agent can learn behaviors to achieve its goal.
To talk more specifically what RL does, we need to introduce additional terminology. We need to talk about
 states and observations,
 action spaces,
 policies,
 trajectories,
 different formulations of return,
 the RL optimization problem,
 and value functions.
States and ObservationsÂ¶
A state is a complete description of the state of the world. There is no information about the world which is hidden from the state. An observation is a partial description of a state, which may omit information.
In deep RL, we almost always represent states and observations by a realvalued vector, matrix, or higherorder tensor. For instance, a visual observation could be represented by the RGB matrix of its pixel values; the state of a robot might be represented by its joint angles and velocities.
When the agent is able to observe the complete state of the environment, we say that the environment is fully observed. When the agent can only see a partial observation, we say that the environment is partially observed.
You Should Know
Reinforcement learning notation sometimes puts the symbol for state, , in places where it would be technically more appropriate to write the symbol for observation, . Specifically, this happens when talking about how the agent decides an action: we often signal in notation that the action is conditioned on the state, when in practice, the action is conditioned on the observation because the agent does not have access to the state.
In our guide, we’ll follow standard conventions for notation, but it should be clear from context which is meant. If something is unclear, though, please raise an issue! Our goal is to teach, not to confuse.
Action SpacesÂ¶
Different environments allow different kinds of actions. The set of all valid actions in a given environment is often called the action space. Some environments, like Atari and Go, have discrete action spaces, where only a finite number of moves are available to the agent. Other environments, like where the agent controls a robot in a physical world, have continuous action spaces. In continuous spaces, actions are realvalued vectors.
This distinction has some quiteprofound consequences for methods in deep RL. Some families of algorithms can only be directly applied in one case, and would have to be substantially reworked for the other.
PoliciesÂ¶
A policy is a rule used by an agent to decide what actions to take. It can be deterministic, in which case it is usually denoted by :
or it may be stochastic, in which case it is usually denoted by :
Because the policy is essentially the agent’s brain, it’s not uncommon to substitute the word “policy” for “agent”, eg saying “The policy is trying to maximize reward.”
In deep RL, we deal with parameterized policies: policies whose outputs are computable functions that depend on a set of parameters (eg the weights and biases of a neural network) which we can adjust to change the behavior via some optimization algorithm.
We often denote the parameters of such a policy by or , and then write this as a subscript on the policy symbol to highlight the connection:
Deterministic PoliciesÂ¶
Example: Deterministic Policies. Here is a code snippet for building a simple deterministic policy for a continuous action space in PyTorch, using the torch.nn
package:
pi_net = nn.Sequential(
nn.Linear(obs_dim, 64),
nn.Tanh(),
nn.Linear(64, 64),
nn.Tanh(),
nn.Linear(64, act_dim)
)
This builds a multilayer perceptron (MLP) network with two hidden layers of size 64 and activation functions. If obs
is a Numpy array containing a batch of observations, pi_net
can be used to obtain a batch of actions as follows:
obs_tensor = torch.as_tensor(obs, dtype=torch.float32)
actions = pi_net(obs_tensor)
You Should Know
Don’t worry about it if this neural network stuff is unfamiliar to you—this tutorial will focus on RL, and not on the neural network side of things. So you can skip this example and come back to it later. But we figured that if you already knew, it could be helpful.
Stochastic PoliciesÂ¶
The two most common kinds of stochastic policies in deep RL are categorical policies and diagonal Gaussian policies.
Categorical policies can be used in discrete action spaces, while diagonal Gaussian policies are used in continuous action spaces.
Two key computations are centrally important for using and training stochastic policies:
 sampling actions from the policy,
 and computing log likelihoods of particular actions, .
In what follows, we’ll describe how to do these for both categorical and diagonal Gaussian policies.
Categorical Policies
A categorical policy is like a classifier over discrete actions. You build the neural network for a categorical policy the same way you would for a classifier: the input is the observation, followed by some number of layers (possibly convolutional or denselyconnected, depending on the kind of input), and then you have one final linear layer that gives you logits for each action, followed by a softmax to convert the logits into probabilities.
Sampling. Given the probabilities for each action, frameworks like PyTorch and Tensorflow have builtin tools for sampling. For example, see the documentation for Categorical distributions in PyTorch, torch.multinomial, tf.distributions.Categorical, or tf.multinomial.
LogLikelihood. Denote the last layer of probabilities as . It is a vector with however many entries as there are actions, so we can treat the actions as indices for the vector. The log likelihood for an action can then be obtained by indexing into the vector:
Diagonal Gaussian Policies
A multivariate Gaussian distribution (or multivariate normal distribution, if you prefer) is described by a mean vector, , and a covariance matrix, . A diagonal Gaussian distribution is a special case where the covariance matrix only has entries on the diagonal. As a result, we can represent it by a vector.
A diagonal Gaussian policy always has a neural network that maps from observations to mean actions, . There are two different ways that the covariance matrix is typically represented.
The first way: There is a single vector of log standard deviations, , which is not a function of state: the are standalone parameters. (You Should Know: our implementations of VPG, TRPO, and PPO do it this way.)
The second way: There is a neural network that maps from states to log standard deviations, . It may optionally share some layers with the mean network.
Note that in both cases we output log standard deviations instead of standard deviations directly. This is because log stds are free to take on any values in , while stds must be nonnegative. It’s easier to train parameters if you don’t have to enforce those kinds of constraints. The standard deviations can be obtained immediately from the log standard deviations by exponentiating them, so we do not lose anything by representing them this way.
Sampling. Given the mean action and standard deviation , and a vector of noise from a spherical Gaussian (), an action sample can be computed with
where denotes the elementwise product of two vectors. Standard frameworks have builtin ways to generate the noise vectors, such as torch.normal or tf.random_normal. Alternatively, you can build distribution objects, eg through torch.distributions.Normal or tf.distributions.Normal, and use them to generate samples. (The advantage of the latter approach is that those objects can also calculate loglikelihoods for you.)
LogLikelihood. The loglikelihood of a dimensional action , for a diagonal Gaussian with mean and standard deviation , is given by
TrajectoriesÂ¶
A trajectory is a sequence of states and actions in the world,
The very first state of the world, , is randomly sampled from the startstate distribution, sometimes denoted by :
State transitions (what happens to the world between the state at time , , and the state at , ), are governed by the natural laws of the environment, and depend on only the most recent action, . They can be either deterministic,
or stochastic,
Actions come from an agent according to its policy.
You Should Know
Trajectories are also frequently called episodes or rollouts.
Reward and ReturnÂ¶
The reward function is critically important in reinforcement learning. It depends on the current state of the world, the action just taken, and the next state of the world:
although frequently this is simplified to just a dependence on the current state, , or stateaction pair .
The goal of the agent is to maximize some notion of cumulative reward over a trajectory, but this actually can mean a few things. We’ll notate all of these cases with , and it will either be clear from context which case we mean, or it won’t matter (because the same equations will apply to all cases).
One kind of return is the finitehorizon undiscounted return, which is just the sum of rewards obtained in a fixed window of steps:
Another kind of return is the infinitehorizon discounted return, which is the sum of all rewards ever obtained by the agent, but discounted by how far off in the future they’re obtained. This formulation of reward includes a discount factor :
Why would we ever want a discount factor, though? Don’t we just want to get all rewards? We do, but the discount factor is both intuitively appealing and mathematically convenient. On an intuitive level: cash now is better than cash later. Mathematically: an infinitehorizon sum of rewards may not converge to a finite value, and is hard to deal with in equations. But with a discount factor and under reasonable conditions, the infinite sum converges.
You Should Know
While the line between these two formulations of return are quite stark in RL formalism, deep RL practice tends to blur the line a fair bit—for instance, we frequently set up algorithms to optimize the undiscounted return, but use discount factors in estimating value functions.
The RL ProblemÂ¶
Whatever the choice of return measure (whether infinitehorizon discounted, or finitehorizon undiscounted), and whatever the choice of policy, the goal in RL is to select a policy which maximizes expected return when the agent acts according to it.
To talk about expected return, we first have to talk about probability distributions over trajectories.
Let’s suppose that both the environment transitions and the policy are stochastic. In this case, the probability of a step trajectory is:
The expected return (for whichever measure), denoted by , is then:
The central optimization problem in RL can then be expressed by
with being the optimal policy.
Value FunctionsÂ¶
It’s often useful to know the value of a state, or stateaction pair. By value, we mean the expected return if you start in that state or stateaction pair, and then act according to a particular policy forever after. Value functions are used, one way or another, in almost every RL algorithm.
There are four main functions of note here.
The OnPolicy Value Function, , which gives the expected return if you start in state and always act according to policy :
The OnPolicy ActionValue Function, , which gives the expected return if you start in state , take an arbitrary action (which may not have come from the policy), and then forever after act according to policy :
The Optimal Value Function, , which gives the expected return if you start in state and always act according to the optimal policy in the environment:
The Optimal ActionValue Function, , which gives the expected return if you start in state , take an arbitrary action , and then forever after act according to the optimal policy in the environment:
You Should Know
When we talk about value functions, if we do not make reference to timedependence, we only mean expected infinitehorizon discounted return. Value functions for finitehorizon undiscounted return would need to accept time as an argument. Can you think about why? Hint: what happens when time’s up?
You Should Know
There are two key connections between the value function and the actionvalue function that come up pretty often:
and
These relations follow pretty directly from the definitions just given: can you prove them?
The Optimal QFunction and the Optimal ActionÂ¶
There is an important connection between the optimal actionvalue function and the action selected by the optimal policy. By definition, gives the expected return for starting in state , taking (arbitrary) action , and then acting according to the optimal policy forever after.
The optimal policy in will select whichever action maximizes the expected return from starting in . As a result, if we have , we can directly obtain the optimal action, , via
Note: there may be multiple actions which maximize , in which case, all of them are optimal, and the optimal policy may randomly select any of them. But there is always an optimal policy which deterministically selects an action.
Bellman EquationsÂ¶
All four of the value functions obey special selfconsistency equations called Bellman equations. The basic idea behind the Bellman equations is this:
The value of your starting point is the reward you expect to get from being there, plus the value of wherever you land next.
The Bellman equations for the onpolicy value functions are
where is shorthand for , indicating that the next state is sampled from the environment’s transition rules; is shorthand for ; and is shorthand for .
The Bellman equations for the optimal value functions are
The crucial difference between the Bellman equations for the onpolicy value functions and the optimal value functions, is the absence or presence of the over actions. Its inclusion reflects the fact that whenever the agent gets to choose its action, in order to act optimally, it has to pick whichever action leads to the highest value.
You Should Know
The term “Bellman backup” comes up quite frequently in the RL literature. The Bellman backup for a state, or stateaction pair, is the righthand side of the Bellman equation: the rewardplusnextvalue.
Advantage FunctionsÂ¶
Sometimes in RL, we don’t need to describe how good an action is in an absolute sense, but only how much better it is than others on average. That is to say, we want to know the relative advantage of that action. We make this concept precise with the advantage function.
The advantage function corresponding to a policy describes how much better it is to take a specific action in state , over randomly selecting an action according to , assuming you act according to forever after. Mathematically, the advantage function is defined by
You Should Know
We’ll discuss this more later, but the advantage function is crucially important to policy gradient methods.
(Optional) FormalismÂ¶
So far, we’ve discussed the agent’s environment in an informal way, but if you try to go digging through the literature, you’re likely to run into the standard mathematical formalism for this setting: Markov Decision Processes (MDPs). An MDP is a 5tuple, , where
 is the set of all valid states,
 is the set of all valid actions,
 is the reward function, with ,
 is the transition probability function, with being the probability of transitioning into state if you start in state and take action ,
 and is the starting state distribution.
The name Markov Decision Process refers to the fact that the system obeys the Markov property: transitions only depend on the most recent state and action, and no prior history.
Part 2: Kinds of RL AlgorithmsÂ¶
Table of Contents
Now that we’ve gone through the basics of RL terminology and notation, we can cover a little bit of the richer material: the landscape of algorithms in modern RL, and a description of the kinds of tradeoffs that go into algorithm design.
A Taxonomy of RL AlgorithmsÂ¶
We’ll start this section with a disclaimer: it’s really quite hard to draw an accurate, allencompassing taxonomy of algorithms in the modern RL space, because the modularity of algorithms is not wellrepresented by a tree structure. Also, to make something that fits on a page and is reasonably digestible in an introduction essay, we have to omit quite a bit of more advanced material (exploration, transfer learning, meta learning, etc). That said, our goals here are
 to highlight the most foundational design choices in deep RL algorithms about what to learn and how to learn it,
 to expose the tradeoffs in those choices,
 and to place a few prominent modern algorithms into context with respect to those choices.
ModelFree vs ModelBased RLÂ¶
One of the most important branching points in an RL algorithm is the question of whether the agent has access to (or learns) a model of the environment. By a model of the environment, we mean a function which predicts state transitions and rewards.
The main upside to having a model is that it allows the agent to plan by thinking ahead, seeing what would happen for a range of possible choices, and explicitly deciding between its options. Agents can then distill the results from planning ahead into a learned policy. A particularly famous example of this approach is AlphaZero. When this works, it can result in a substantial improvement in sample efficiency over methods that don’t have a model.
The main downside is that a groundtruth model of the environment is usually not available to the agent. If an agent wants to use a model in this case, it has to learn the model purely from experience, which creates several challenges. The biggest challenge is that bias in the model can be exploited by the agent, resulting in an agent which performs well with respect to the learned model, but behaves suboptimally (or super terribly) in the real environment. Modellearning is fundamentally hard, so even intense effort—being willing to throw lots of time and compute at it—can fail to pay off.
Algorithms which use a model are called modelbased methods, and those that don’t are called modelfree. While modelfree methods forego the potential gains in sample efficiency from using a model, they tend to be easier to implement and tune. As of the time of writing this introduction (September 2018), modelfree methods are more popular and have been more extensively developed and tested than modelbased methods.
What to LearnÂ¶
Another critical branching point in an RL algorithm is the question of what to learn. The list of usual suspects includes
 policies, either stochastic or deterministic,
 actionvalue functions (Qfunctions),
 value functions,
 and/or environment models.
What to Learn in ModelFree RLÂ¶
There are two main approaches to representing and training agents with modelfree RL:
Policy Optimization. Methods in this family represent a policy explicitly as . They optimize the parameters either directly by gradient ascent on the performance objective , or indirectly, by maximizing local approximations of . This optimization is almost always performed onpolicy, which means that each update only uses data collected while acting according to the most recent version of the policy. Policy optimization also usually involves learning an approximator for the onpolicy value function , which gets used in figuring out how to update the policy.
A couple of examples of policy optimization methods are:
 A2C / A3C, which performs gradient ascent to directly maximize performance,
 and PPO, whose updates indirectly maximize performance, by instead maximizing a surrogate objective function which gives a conservative estimate for how much will change as a result of the update.
QLearning. Methods in this family learn an approximator for the optimal actionvalue function, . Typically they use an objective function based on the Bellman equation. This optimization is almost always performed offpolicy, which means that each update can use data collected at any point during training, regardless of how the agent was choosing to explore the environment when the data was obtained. The corresponding policy is obtained via the connection between and : the actions taken by the Qlearning agent are given by
Examples of Qlearning methods include
 DQN, a classic which substantially launched the field of deep RL,
 and C51, a variant that learns a distribution over return whose expectation is .
Tradeoffs Between Policy Optimization and QLearning. The primary strength of policy optimization methods is that they are principled, in the sense that you directly optimize for the thing you want. This tends to make them stable and reliable. By contrast, Qlearning methods only indirectly optimize for agent performance, by training to satisfy a selfconsistency equation. There are many failure modes for this kind of learning, so it tends to be less stable. [1] But, Qlearning methods gain the advantage of being substantially more sample efficient when they do work, because they can reuse data more effectively than policy optimization techniques.
Interpolating Between Policy Optimization and QLearning. Serendipitously, policy optimization and Qlearning are not incompatible (and under some circumstances, it turns out, equivalent), and there exist a range of algorithms that live in between the two extremes. Algorithms that live on this spectrum are able to carefully tradeoff between the strengths and weaknesses of either side. Examples include
 DDPG, an algorithm which concurrently learns a deterministic policy and a Qfunction by using each to improve the other,
 and SAC, a variant which uses stochastic policies, entropy regularization, and a few other tricks to stabilize learning and score higher than DDPG on standard benchmarks.
[1]  For more information about how and why Qlearning methods can fail, see 1) this classic paper by Tsitsiklis and van Roy, 2) the (much more recent) review by Szepesvari (in section 4.3.2), and 3) chapter 11 of Sutton and Barto, especially section 11.3 (on “the deadly triad” of function approximation, bootstrapping, and offpolicy data, together causing instability in valuelearning algorithms). 
What to Learn in ModelBased RLÂ¶
Unlike modelfree RL, there aren’t a small number of easytodefine clusters of methods for modelbased RL: there are many orthogonal ways of using models. We’ll give a few examples, but the list is far from exhaustive. In each case, the model may either be given or learned.
Background: Pure Planning. The most basic approach never explicitly represents the policy, and instead, uses pure planning techniques like modelpredictive control (MPC) to select actions. In MPC, each time the agent observes the environment, it computes a plan which is optimal with respect to the model, where the plan describes all actions to take over some fixed window of time after the present. (Future rewards beyond the horizon may be considered by the planning algorithm through the use of a learned value function.) The agent then executes the first action of the plan, and immediately discards the rest of it. It computes a new plan each time it prepares to interact with the environment, to avoid using an action from a plan with a shorterthandesired planning horizon.
 The MBMF work explores MPC with learned environment models on some standard benchmark tasks for deep RL.
Expert Iteration. A straightforward followon to pure planning involves using and learning an explicit representation of the policy, . The agent uses a planning algorithm (like Monte Carlo Tree Search) in the model, generating candidate actions for the plan by sampling from its current policy. The planning algorithm produces an action which is better than what the policy alone would have produced, hence it is an “expert” relative to the policy. The policy is afterwards updated to produce an action more like the planning algorithm’s output.
 The ExIt algorithm uses this approach to train deep neural networks to play Hex.
 AlphaZero is another example of this approach.
Data Augmentation for ModelFree Methods. Use a modelfree RL algorithm to train a policy or Qfunction, but either 1) augment real experiences with fictitious ones in updating the agent, or 2) use only fictitous experience for updating the agent.
 See MBVE for an example of augmenting real experiences with fictitious ones.
 See World Models for an example of using purely fictitious experience to train the agent, which they call “training in the dream.”
Embedding Planning Loops into Policies. Another approach embeds the planning procedure directly into a policy as a subroutine—so that complete plans become side information for the policy—while training the output of the policy with any standard modelfree algorithm. The key concept is that in this framework, the policy can learn to choose how and when to use the plans. This makes model bias less of a problem, because if the model is bad for planning in some states, the policy can simply learn to ignore it.
 See I2A for an example of agents being endowed with this style of imagination.
Links to Algorithms in TaxonomyÂ¶
[2]  A2C / A3C (Asynchronous Advantage ActorCritic): Mnih et al, 2016 
[3]  PPO (Proximal Policy Optimization): Schulman et al, 2017 
[4]  TRPO (Trust Region Policy Optimization): Schulman et al, 2015 
[5]  DDPG (Deep Deterministic Policy Gradient): Lillicrap et al, 2015 
[6]  TD3 (Twin Delayed DDPG): Fujimoto et al, 2018 
[7]  SAC (Soft ActorCritic): Haarnoja et al, 2018 
[8]  DQN (Deep QNetworks): Mnih et al, 2013 
[9]  C51 (Categorical 51Atom DQN): Bellemare et al, 2017 
[10]  QRDQN (Quantile Regression DQN): Dabney et al, 2017 
[11]  HER (Hindsight Experience Replay): Andrychowicz et al, 2017 
[12]  World Models: Ha and Schmidhuber, 2018 
[13]  I2A (ImaginationAugmented Agents): Weber et al, 2017 
[14]  MBMF (ModelBased RL with ModelFree FineTuning): Nagabandi et al, 2017 
[15]  MBVE (ModelBased Value Expansion): Feinberg et al, 2018 
[16]  AlphaZero: Silver et al, 2017 
Part 3: Intro to Policy OptimizationÂ¶
Table of Contents
In this section, we’ll discuss the mathematical foundations of policy optimization algorithms, and connect the material to sample code. We will cover three key results in the theory of policy gradients:
 the simplest equation describing the gradient of policy performance with respect to policy parameters,
 a rule which allows us to drop useless terms from that expression,
 and a rule which allows us to add useful terms to that expression.
In the end, we’ll tie those results together and describe the advantagebased expression for the policy gradient—the version we use in our Vanilla Policy Gradient implementation.
Deriving the Simplest Policy GradientÂ¶
Here, we consider the case of a stochastic, parameterized policy, . We aim to maximize the expected return . For the purposes of this derivation, we’ll take to give the finitehorizon undiscounted return, but the derivation for the infinitehorizon discounted return setting is almost identical.
We would like to optimize the policy by gradient ascent, eg
The gradient of policy performance, , is called the policy gradient, and algorithms that optimize the policy this way are called policy gradient algorithms. (Examples include Vanilla Policy Gradient and TRPO. PPO is often referred to as a policy gradient algorithm, though this is slightly inaccurate.)
To actually use this algorithm, we need an expression for the policy gradient which we can numerically compute. This involves two steps: 1) deriving the analytical gradient of policy performance, which turns out to have the form of an expected value, and then 2) forming a sample estimate of that expected value, which can be computed with data from a finite number of agentenvironment interaction steps.
In this subsection, we’ll find the simplest form of that expression. In later subsections, we’ll show how to improve on the simplest form to get the version we actually use in standard policy gradient implementations.
We’ll begin by laying out a few facts which are useful for deriving the analytical gradient.
1. Probability of a Trajectory. The probability of a trajectory given that actions come from is
2. The LogDerivative Trick. The logderivative trick is based on a simple rule from calculus: the derivative of with respect to is . When rearranged and combined with chain rule, we get:
3. LogProbability of a Trajectory. The logprob of a trajectory is just
4. Gradients of Environment Functions. The environment has no dependence on , so gradients of , , and are zero.
5. GradLogProb of a Trajectory. The gradient of the logprob of a trajectory is thus
Putting it all together, we derive the following:
Derivation for Basic Policy Gradient
This is an expectation, which means that we can estimate it with a sample mean. If we collect a set of trajectories where each trajectory is obtained by letting the agent act in the environment using the policy , the policy gradient can be estimated with
where is the number of trajectories in (here, ).
This last expression is the simplest version of the computable expression we desired. Assuming that we have represented our policy in a way which allows us to calculate , and if we are able to run the policy in the environment to collect the trajectory dataset, we can compute the policy gradient and take an update step.
Implementing the Simplest Policy GradientÂ¶
We give a short PyTorch implementation of this simple version of the policy gradient algorithm in spinup/examples/pytorch/pg_math/1_simple_pg.py
. (It can also be viewed on github.) It is only 128 lines long, so we highly recommend reading through it in depth. While we won’t go through the entirety of the code here, we’ll highlight and explain a few important pieces.
You Should Know
This section was previously written with a Tensorflow example. The old Tensorflow section can be found here.
1. Making the Policy Network.
30 31 32 33 34 35 36 37 38 39 40  # make core of policy network
logits_net = mlp(sizes=[obs_dim]+hidden_sizes+[n_acts])
# make function to compute action distribution
def get_policy(obs):
logits = logits_net(obs)
return Categorical(logits=logits)
# make action selection function (outputs int actions, sampled from policy)
def get_action(obs):
return get_policy(obs).sample().item()

This block builds modules and functions for using a feedforward neural network categorical policy. (See the Stochastic Policies section in Part 1 for a refresher.) The output from the logits_net
module can be used to construct logprobabilities and probabilities for actions, and the get_action
function samples actions based on probabilities computed from the logits. (Note: this particular get_action
function assumes that there will only be one obs
provided, and therefore only one integer action output. That’s why it uses .item()
, which is used to get the contents of a Tensor with only one element.)
A lot of work in this example is getting done by the Categorical
object on L36. This is a PyTorch Distribution
object that wraps up some mathematical functions associated with probability distributions. In particular, it has a method for sampling from the distribution (which we use on L40) and a method for computing log probabilities of given samples (which we use later). Since PyTorch distributions are really useful for RL, check out their documentation to get a feel for how they work.
You Should Know
Friendly reminder! When we talk about a categorical distribution having “logits,” what we mean is that the probabilities for each outcome are given by the Softmax function of the logits. That is, the probability for action under a categorical distribution with logits is:
2. Making the Loss Function.
42 43 44 45  # make loss function whose gradient, for the right data, is policy gradient
def compute_loss(obs, act, weights):
logp = get_policy(obs).log_prob(act)
return (logp * weights).mean()

In this block, we build a “loss” function for the policy gradient algorithm. When the right data is plugged in, the gradient of this loss is equal to the policy gradient. The right data means a set of (state, action, weight) tuples collected while acting according to the current policy, where the weight for a stateaction pair is the return from the episode to which it belongs. (Although as we will show in later subsections, there are other values you can plug in for the weight which also work correctly.)
You Should Know
Even though we describe this as a loss function, it is not a loss function in the typical sense from supervised learning. There are two main differences from standard loss functions.
1. The data distribution depends on the parameters. A loss function is usually defined on a fixed data distribution which is independent of the parameters we aim to optimize. Not so here, where the data must be sampled on the most recent policy.
2. It doesn’t measure performance. A loss function usually evaluates the performance metric that we care about. Here, we care about expected return, , but our “loss” function does not approximate this at all, even in expectation. This “loss” function is only useful to us because, when evaluated at the current parameters, with data generated by the current parameters, it has the negative gradient of performance.
But after that first step of gradient descent, there is no more connection to performance. This means that minimizing this “loss” function, for a given batch of data, has no guarantee whatsoever of improving expected return. You can send this loss to and policy performance could crater; in fact, it usually will. Sometimes a deep RL researcher might describe this outcome as the policy “overfitting” to a batch of data. This is descriptive, but should not be taken literally because it does not refer to generalization error.
We raise this point because it is common for ML practitioners to interpret a loss function as a useful signal during training—”if the loss goes down, all is well.” In policy gradients, this intuition is wrong, and you should only care about average return. The loss function means nothing.
You Should Know
The approach used here to make the logp
tensor–calling the log_prob
method of a PyTorch Categorical
object–may require some modification to work with other kinds of distribution objects.
For example, if you are using a Normal distribution (for a diagonal Gaussian policy), the output from calling policy.log_prob(act)
will give you a Tensor containing separate log probabilities for each component of each vectorvalued action. That is to say, you put in a Tensor of shape (batch, act_dim)
, and get out a Tensor of shape (batch, act_dim)
, when what you need for making an RL loss is a Tensor of shape (batch,)
. In that case, you would sum up the log probabilities of the action components to get the log probabilities of the actions. That is, you would compute:
logp = get_policy(obs).log_prob(act).sum(axis=1)
3. Running One Epoch of Training.
50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112  # for training policy
def train_one_epoch():
# make some empty lists for logging.
batch_obs = [] # for observations
batch_acts = [] # for actions
batch_weights = [] # for R(tau) weighting in policy gradient
batch_rets = [] # for measuring episode returns
batch_lens = [] # for measuring episode lengths
# reset episodespecific variables
obs = env.reset() # first obs comes from starting distribution
done = False # signal from environment that episode is over
ep_rews = [] # list for rewards accrued throughout ep
# render first episode of each epoch
finished_rendering_this_epoch = False
# collect experience by acting in the environment with current policy
while True:
# rendering
if (not finished_rendering_this_epoch) and render:
env.render()
# save obs
batch_obs.append(obs.copy())
# act in the environment
act = get_action(torch.as_tensor(obs, dtype=torch.float32))
obs, rew, done, _ = env.step(act)
# save action, reward
batch_acts.append(act)
ep_rews.append(rew)
if done:
# if episode is over, record info about episode
ep_ret, ep_len = sum(ep_rews), len(ep_rews)
batch_rets.append(ep_ret)
batch_lens.append(ep_len)
# the weight for each logprob(as) is R(tau)
batch_weights += [ep_ret] * ep_len
# reset episodespecific variables
obs, done, ep_rews = env.reset(), False, []
# won't render again this epoch
finished_rendering_this_epoch = True
# end experience loop if we have enough of it
if len(batch_obs) > batch_size:
break
# take a single policy gradient update step
optimizer.zero_grad()
batch_loss = compute_loss(obs=torch.as_tensor(batch_obs, dtype=torch.float32),
act=torch.as_tensor(batch_acts, dtype=torch.int32),
weights=torch.as_tensor(batch_weights, dtype=torch.float32)
)
batch_loss.backward()
optimizer.step()
return batch_loss, batch_rets, batch_lens

The train_one_epoch()
function runs one “epoch” of policy gradient, which we define to be
 the experience collection step (L67102), where the agent acts for some number of episodes in the environment using the most recent policy, followed by
 a single policy gradient update step (L104111).
The main loop of the algorithm just repeatedly calls train_one_epoch()
.
You Should Know
If you aren’t already familiar with optimization in PyTorch, observe the pattern for taking one gradient descent step as shown in lines 104111. First, clear the gradient buffers. Then, compute the loss function. Then, compute a backward pass on the loss function; this accumulates fresh gradients into the gradient buffers. Finally, take a step with the optimizer.
Expected GradLogProb LemmaÂ¶
In this subsection, we will derive an intermediate result which is extensively used throughout the theory of policy gradients. We will call it the Expected GradLogProb (EGLP) lemma. [1]
EGLP Lemma. Suppose that is a parameterized probability distribution over a random variable, . Then:
Proof
Recall that all probability distributions are normalized:
Take the gradient of both sides of the normalization condition:
Use the log derivative trick to get:
[1]  The author of this article is not aware of this lemma being given a standard name anywhere in the literature. But given how often it comes up, it seems pretty worthwhile to give it some kind of name for ease of reference. 
Don’t Let the Past Distract YouÂ¶
Examine our most recent expression for the policy gradient:
Taking a step with this gradient pushes up the logprobabilities of each action in proportion to , the sum of all rewards ever obtained. But this doesn’t make much sense.
Agents should really only reinforce actions on the basis of their consequences. Rewards obtained before taking an action have no bearing on how good that action was: only rewards that come after.
It turns out that this intuition shows up in the math, and we can show that the policy gradient can also be expressed by
In this form, actions are only reinforced based on rewards obtained after they are taken.
We’ll call this form the “rewardtogo policy gradient,” because the sum of rewards after a point in a trajectory,
is called the rewardtogo from that point, and this policy gradient expression depends on the rewardtogo from stateaction pairs.
You Should Know
But how is this better? A key problem with policy gradients is how many sample trajectories are needed to get a lowvariance sample estimate for them. The formula we started with included terms for reinforcing actions proportional to past rewards, all of which had zero mean, but nonzero variance: as a result, they would just add noise to sample estimates of the policy gradient. By removing them, we reduce the number of sample trajectories needed.
An (optional) proof of this claim can be found `here`_, and it ultimately depends on the EGLP lemma.
Implementing RewardtoGo Policy GradientÂ¶
We give a short PyTorch implementation of the rewardtogo policy gradient in spinup/examples/pytorch/pg_math/2_rtg_pg.py
. (It can also be viewed on github.)
The only thing that has changed from 1_simple_pg.py
is that we now use different weights in the loss function. The code modification is very slight: we add a new function, and change two other lines. The new function is:
17 18 19 20 21 22  def reward_to_go(rews):
n = len(rews)
rtgs = np.zeros_like(rews)
for i in reversed(range(n)):
rtgs[i] = rews[i] + (rtgs[i+1] if i+1 < n else 0)
return rtgs

And then we tweak the old L9192 from:
91 92  # the weight for each logprob(as) is R(tau)
batch_weights += [ep_ret] * ep_len

to:
98 99  # the weight for each logprob(a_ts_t) is rewardtogo from t
batch_weights += list(reward_to_go(ep_rews))

You Should Know
This section was previously written with a Tensorflow example. The old Tensorflow section can be found here.
Baselines in Policy GradientsÂ¶
An immediate consequence of the EGLP lemma is that for any function which only depends on state,
This allows us to add or subtract any number of terms like this from our expression for the policy gradient, without changing it in expectation:
Any function used in this way is called a baseline.
The most common choice of baseline is the onpolicy value function . Recall that this is the average return an agent gets if it starts in state and then acts according to policy for the rest of its life.
Empirically, the choice has the desirable effect of reducing variance in the sample estimate for the policy gradient. This results in faster and more stable policy learning. It is also appealing from a conceptual angle: it encodes the intuition that if an agent gets what it expected, it should “feel” neutral about it.
You Should Know
In practice, cannot be computed exactly, so it has to be approximated. This is usually done with a neural network, , which is updated concurrently with the policy (so that the value network always approximates the value function of the most recent policy).
The simplest method for learning , used in most implementations of policy optimization algorithms (including VPG, TRPO, PPO, and A2C), is to minimize a meansquarederror objective:
where is the policy at epoch . This is done with one or more steps of gradient descent, starting from the previous value parameters .
Other Forms of the Policy GradientÂ¶
What we have seen so far is that the policy gradient has the general form
where could be any of
or
or
All of these choices lead to the same expected value for the policy gradient, despite having different variances. It turns out that there are two more valid choices of weights which are important to know.
1. OnPolicy ActionValue Function. The choice
is also valid. See this page for an (optional) proof of this claim.
2. The Advantage Function. Recall that the advantage of an action, defined by , describes how much better or worse it is than other actions on average (relative to the current policy). This choice,
is also valid. The proof is that it’s equivalent to using and then using a value function baseline, which we are always free to do.
You Should Know
The formulation of policy gradients with advantage functions is extremely common, and there are many different ways of estimating the advantage function used by different algorithms.
You Should Know
For a more detailed treatment of this topic, you should read the paper on Generalized Advantage Estimation (GAE), which goes into depth about different choices of in the background sections.
That paper then goes on to describe GAE, a method for approximating the advantage function in policy optimization algorithms which enjoys widespread use. For instance, Spinning Up’s implementations of VPG, TRPO, and PPO make use of it. As a result, we strongly advise you to study it.
RecapÂ¶
In this chapter, we described the basic theory of policy gradient methods and connected some of the early results to code examples. The interested student should continue from here by studying how the later results (value function baselines and the advantage formulation of policy gradients) translate into Spinning Up’s implementation of Vanilla Policy Gradient.
Spinning Up as a Deep RL ResearcherÂ¶
By Joshua Achiam, October 13th, 2018
Table of Contents
If youâ€™re an aspiring deep RL researcher, youâ€™ve probably heard all kinds of things about deep RL by this point. You know that itâ€™s hard and it doesnâ€™t always work. That even when youâ€™re following a recipe, reproducibility is a challenge. And that if youâ€™re starting from scratch, the learning curve is incredibly steep. Itâ€™s also the case that there are a lot of great resources out there, but the material is new enough that thereâ€™s not a clear, wellcharted path to mastery. The goal of this column is to help you get past the initial hurdle, and give you a clear sense of how to spin up as a deep RL researcher. In particular, this will outline a useful curriculum for increasing raw knowledge, while interleaving it with the odds and ends that lead to better research.
The Right BackgroundÂ¶
Build up a solid mathematical background. From probability and statistics, feel comfortable with random variables, Bayesâ€™ theorem, chain rule of probability, expected values, standard deviations, and importance sampling. From multivariate calculus, understand gradients and (optionally, but itâ€™ll help) Taylor series expansions.
Build up a general knowledge of deep learning. You donâ€™t need to know every single special trick and architecture, but the basics help. Know about standard architectures (MLP, vanilla RNN, LSTM (also see this blog), GRU, conv layers, resnets, attention mechanisms), common regularizers (weight decay, dropout), normalization (batch norm, layer norm, weight norm), and optimizers (SGD, momentum SGD, Adam, others). Know what the reparameterization trick is.
Become familiar with at least one deep learning library. Tensorflow or PyTorch would be a good place to start. You donâ€™t need to know how to do everything, but you should feel pretty confident in implementing a simple program to do supervised learning.
Get comfortable with the main concepts and terminology in RL. Know what states, actions, trajectories, policies, rewards, value functions, and actionvalue functions are. If you’re unfamiliar, Spinning Up ships with an introduction to this material; it’s also worth checking out the RLIntro from the OpenAI Hackathon, or the exceptional and thorough overview by Lilian Weng. Optionally, if youâ€™re the sort of person who enjoys mathematical theory, study up on the math of monotonic improvement theory (which forms the basis for advanced policy gradient algorithms), or classical RL algorithms (which despite being superseded by deep RL algorithms, contain valuable insights that sometimes drive new research).
Learn by DoingÂ¶
Write your own implementations. You should implement as many of the core deep RL algorithms from scratch as you can, with the aim of writing the shortest correct implementation of each. This is by far the best way to develop an understanding of how they work, as well as intuitions for their specific performance characteristics.
Simplicity is critical. You should organize your efforts so that you implement the simplest algorithms first, and only gradually introduce complexity. If you start off trying to build something with too many moving parts, odds are good that it will break and you’ll lose weeks trying to debug it. This is a common failure mode for people who are new to deep RL, and if you find yourself stuck in it, don’t be discouraged—but do try to change tack and work on a simpler algorithm instead, before returning to the more complex thing later.
Which algorithms? You should probably start with vanilla policy gradient (also called REINFORCE), DQN, A2C (the synchronous version of A3C), PPO (the variant with the clipped objective), and DDPG, approximately in that order. The simplest versions of all of these can be written in just a few hundred lines of code (ballpark 250300), and some of them even less (for example, a nofrills version of VPG can be written in about 80 lines). Write singlethreaded code before you try writing parallelized versions of these algorithms. (Do try to parallelize at least one.)
Focus on understanding. Writing working RL code requires clear, detailoriented understanding of the algorithms. This is because broken RL code almost always fails silently, where the code appears to run fine except that the agent never learns how to solve the task. Usually the problem is that something is being calculated with the wrong equation, or on the wrong distribution, or data is being piped into the wrong place. Sometimes the only way to find these bugs is to read the code with a critical eye, know exactly what it should be doing, and find where it deviates from the correct behavior. Developing that knowledge requires you to engage with both academic literature and other existing implementations (when possible), so a good amount of your time should be spent on that reading.
What to look for in papers: When implementing an algorithm based on a paper, scour that paper, especially the ablation analyses and supplementary material (where available). The ablations will give you an intuition for what parameters or subroutines have the biggest impact on getting things to work, which will help you diagnose bugs. Supplementary material will often give information about specific details like network architecture and optimization hyperparameters, and you should try to align your implementation to these details to improve your chances of getting it working.
But don’t overfit to paper details. Sometimes, the paper prescribes the use of more tricks than are strictly necessary, so be a bit wary of this, and try out simplifications where possible. For example, the original DDPG paper suggests a complex neural network architecture and initialization scheme, as well as batch normalization. These aren’t strictly necessary, and some of the bestreported results for DDPG use simpler networks. As another example, the original A3C paper uses asynchronous updates from the various actorlearners, but it turns out that synchronous updates work about as well.
Don’t overfit to existing implementations either. Study existing implementations for inspiration, but be careful not to overfit to the engineering details of those implementations. RL libraries frequently make choices for abstraction that are good for code reuse between algorithms, but which are unnecessary if you’re only writing a single algorithm or supporting a single use case.
Iterate fast in simple environments. To debug your implementations, try them with simple environments where learning should happen quickly, like CartPolev0, InvertedPendulumv0, FrozenLakev0, and HalfCheetahv2 (with a short time horizon—only 100 or 250 steps instead of the full 1000) from the OpenAI Gym. Donâ€™t try to run an algorithm in Atari or a complex Humanoid environment if you havenâ€™t first verified that it works on the simplest possible toy task. Your ideal experiment turnaroundtime at the debug stage is <5 minutes (on your local machine) or slightly longer but not much. These smallscale experiments don’t require any special hardware, and can be run without too much trouble on CPUs.
If it doesn’t work, assume there’s a bug. Spend a lot of effort searching for bugs before you resort to tweaking hyperparameters: usually itâ€™s a bug. Bad hyperparameters can significantly degrade RL performance, but if you’re using hyperparameters similar to the ones in papers and standard implementations, those will probably not be the issue. Also worth keeping in mind: sometimes things will work in one environment even when you have a breaking bug, so make sure to test in more than one environment once your results look promising.
Measure everything. Do a lot of instrumenting to see whatâ€™s going on underthehood. The more stats about the learning process you read out at each iteration, the easier it is to debug—after all, you canâ€™t tell itâ€™s broken if you canâ€™t see that itâ€™s breaking. I personally like to look at the mean/std/min/max for cumulative rewards, episode lengths, and value function estimates, along with the losses for the objectives, and the details of any exploration parameters (like mean entropy for stochastic policy optimization, or current epsilon for epsilongreedy as in DQN). Also, watch videos of your agentâ€™s performance every now and then; this will give you some insights you wouldnâ€™t get otherwise.
Scale experiments when things work. After you have an implementation of an RL algorithm that seems to work correctly in the simplest environments, test it out on harder environments. Experiments at this stage will take longer—on the order of somewhere between a few hours and a couple of days, depending. Specialized hardware—like a beefy GPU or a 32core machine—might be useful at this point, and you should consider looking into cloud computing resources like AWS or GCE.
Keep these habits! These habits are worth keeping beyond the stage where youâ€™re just learning about deep RL—they will accelerate your research!
Developing a Research ProjectÂ¶
Once you feel reasonably comfortable with the basics in deep RL, you should start pushing on the boundaries and doing research. To get there, you’ll need an idea for a project.
Start by exploring the literature to become aware of topics in the field. There are a wide range of topics you might find interesting: sample efficiency, exploration, transfer learning, hierarchy, memory, modelbased RL, meta learning, and multiagent, to name a few. If you’re looking for inspiration, or just want to get a rough sense of what’s out there, check out Spinning Up’s key papers list. Find a paper that you enjoy on one of these subjects—something that inspires you—and read it thoroughly. Use the related work section and citations to find closelyrelated papers and do a deep dive in the literature. Youâ€™ll start to figure out where the unsolved problems are and where you can make an impact.
Approaches to ideageneration: There are a many different ways to start thinking about ideas for projects, and the frame you choose influences how the project might evolve and what risks it will face. Here are a few examples:
Frame 1: Improving on an Existing Approach. This is the incrementalist angle, where you try to get performance gains in an established problem setting by tweaking an existing algorithm. Reimplementing prior work is super helpful here, because it exposes you to the ways that existing algorithms are brittle and could be improved. A novice will find this the most accessible frame, but it can also be worthwhile for researchers at any level of experience. While some researchers find incrementalism less exciting, some of the most impressive achievements in machine learning have come from work of this nature.
Because projects like these are tied to existing methods, they are by nature narrowly scoped and can wrap up quickly (a few months), which may be desirable (especially when starting out as a researcher). But this also sets up the risks: it’s possible that the tweaks you have in mind for an algorithm may fail to improve it, in which case, unless you come up with more tweaks, the project is just over and you have no clear signal on what to do next.
Frame 2: Focusing on Unsolved Benchmarks. Instead of thinking about how to improve an existing method, you aim to succeed on a task that no one has solved before. For example: achieving perfect generalization from training levels to test levels in the Sonic domain or Gym Retro. When you hammer away at an unsolved task, you might try a wide variety of methods, including prior approaches and new ones that you invent for the project. It is possible for a novice to approch this kind of problem, but there will be a steeper learning curve.
Projects in this frame have a broad scope and can go on for a while (several months to a yearplus). The main risk is that the benchmark is unsolvable without a substantial breakthrough, meaning that it would be easy to spend a lot of time without making any progress on it. But even if a project like this fails, it often leads the researcher to many new insights that become fertile soil for the next project.
Frame 3: Create a New Problem Setting. Instead of thinking about existing methods or current grand challenges, think of an entirely different conceptual problem that hasn’t been studied yet. Then, figure out how to make progress on it. For projects along these lines, a standard benchmark probably doesn’t exist yet, and you will have to design one. This can be a huge challenge, but itâ€™s worth embracing—great benchmarks move the whole field forward.
Problems in this frame come up when they come up—it’s hard to go looking for them.
Avoid reinventing the wheel. When you come up with a good idea that you want to start testing, thatâ€™s great! But while youâ€™re still in the early stages with it, do the most thorough check you can to make sure it hasnâ€™t already been done. It can be pretty disheartening to get halfway through a project, and only then discover that there’s already a paper about your idea. It’s especially frustrating when the work is concurrent, which happens from time to time! But donâ€™t let that deter you—and definitely donâ€™t let it motivate you to plant flags with notquitefinished research and overclaim the merits of the partial work. Do good research and finish out your projects with complete and thorough investigations, because thatâ€™s what counts, and by far what matters most in the long run.
Doing Rigorous Research in RLÂ¶
Now youâ€™ve come up with an idea, and youâ€™re fairly certain it hasnâ€™t been done. You use the skills youâ€™ve developed to implement it and you start testing it out on standard domains. It looks like it works! But what does that mean, and how well does it have to work to be important? This is one of the hardest parts of research in deep RL. In order to validate that your proposal is a meaningful contribution, you have to rigorously prove that it actually gets a performance benefit over the strongest possible baseline algorithm—whatever currently achieves SOTA (state of the art) on your test domains. If youâ€™ve invented a new test domain, so thereâ€™s no previous SOTA, you still need to try out whatever the most reliable algorithm in the literature is that could plausibly do well in the new test domain, and then you have to beat that.
Set up fair comparisons. If you implement your baseline from scratch—as opposed to comparing against another paperâ€™s numbers directly—itâ€™s important to spend as much time tuning your baseline as you spend tuning your own algorithm. This will make sure that comparisons are fair. Also, do your best to hold â€œall else equalâ€ even if there are substantial differences between your algorithm and the baseline. For example, if youâ€™re investigating architecture variants, keep the number of model parameters approximately equal between your model and the baseline. Under no circumstances handicap the baseline! It turns out that the baselines in RL are pretty strong, and getting big, consistent wins over them can be tricky or require some good insight in algorithm design.
Remove stochasticity as a confounder. Beware of random seeds making things look stronger or weaker than they really are, so run everything for many random seeds (at least 3, but if you want to be thorough, do 10 or more). This is really important and deserves a lot of emphasis: deep RL seems fairly brittle with respect to random seed in a lot of common use cases. Thereâ€™s potentially enough variance that two different groups of random seeds can yield learning curves with differences so significant that they look like they donâ€™t come from the same distribution at all (see figure 10 here).
Run highintegrity experiments. Donâ€™t just take the results from the best or most interesting runs to use in your paper. Instead, launch new, final experiments—for all of the methods that you intend to compare (if you are comparing against your own baseline implementations)—and precommit to report on whatever comes out of that. This is to enforce a weak form of preregistration: you use the tuning stage to come up with your hypotheses, and you use the final runs to come up with your conclusions.
Check each claim separately. Another critical aspect of doing research is to run an ablation analysis. Any method you propose is likely to have several key design decisions—like architecture choices or regularization techniques, for instance—each of which could separately impact performance. The claim you’ll make in your work is that those design decisions collectively help, but this is really a bundle of several claims in disguise: one for each such design element. By systematically evaluating what would happen if you were to swap them out with alternate design choices, or remove them entirely, you can figure out how to correctly attribute credit for the benefits your method confers. This lets you make each separate claim with a measure of confidence, and increases the overall strength of your work.
Closing ThoughtsÂ¶
Deep RL is an exciting, fastmoving field, and we need as many people as possible to go through the open problems and make progress on them. Hopefully, you feel a bit more prepared to be a part of it after reading this! And whenever youâ€™re ready, let us know.
PS: Other ResourcesÂ¶
Consider reading through these other informative articles about growing as a researcher or engineer in this field:
Advice for Shortterm Machine Learning Research Projects, by Tim RocktÃ¤schel, Jakob Foerster and Greg Farquhar.
ML Engineering for AI Safety & Robustness: a Google Brain Engineerâ€™s Guide to Entering the Field, by Catherine Olsson and 80,000 Hours.
ReferencesÂ¶
[1]  Deep Reinforcement Learning Doesn’t Work Yet, Alex Irpan, 2018 
[2]  Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control, Islam et al, 2017 
[3]  Deep Reinforcement Learning that Matters, Henderson et al, 2017 
[4]  Lessons Learned Reproducing a Deep Reinforcement Learning Paper, Matthew Rahtz, 2018 
[5]  UCL Course on RL 
[6]  Berkeley Deep RL Course 
[7]  Deep RL Bootcamp 
[8]  Nuts and Bolts of Deep RL, John Schulman 
[9]  Stanford Deep Learning Tutorial: MultiLayer Neural Network 
[10]  The Unreasonable Effectiveness of Recurrent Neural Networks, Andrej Karpathy, 2015 
[11]  LSTM: A Search Space Odyssey, Greff et al, 2015 
[12]  Understanding LSTM Networks, Chris Olah, 2015 
[13]  Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling, Chung et al, 2014 (GRU paper) 
[14]  Conv Nets: A Modular Perspective, Chris Olah, 2014 
[15]  Stanford CS231n, Convolutional Neural Networks for Visual Recognition 
[16]  Deep Residual Learning for Image Recognition, He et al, 2015 (ResNets) 
[17]  Neural Machine Translation by Jointly Learning to Align and Translate, Bahdanau et al, 2014 (Attention mechanisms) 
[18]  Attention Is All You Need, Vaswani et al, 2017 
[19]  A Simple Weight Decay Can Improve Generalization, Krogh and Hertz, 1992 
[20]  Dropout: A Simple Way to Prevent Neural Networks from Overfitting, Srivastava et al, 2014 
[21]  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, Ioffe and Szegedy, 2015 
[22]  Layer Normalization, Ba et al, 2016 
[23]  Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks, Salimans and Kingma, 2016 
[24]  Stanford Deep Learning Tutorial: Stochastic Gradient Descent 
[25]  Adam: A Method for Stochastic Optimization, Kingma and Ba, 2014 
[26]  An overview of gradient descent optimization algorithms, Sebastian Ruder, 2016 
[27]  AutoEncoding Variational Bayes, Kingma and Welling, 2013 (Reparameterization trick) 
[28]  Tensorflow 
[29]  PyTorch 
[30]  Spinning Up in Deep RL: Introduction to RL, Part 1 
[31]  RLIntro Slides from OpenAI Hackathon, Josh Achiam, 2018 
[32]  A (Long) Peek into Reinforcement Learning, Lilian Weng, 2018 
[33]  Optimizing Expectations, John Schulman, 2016 (Monotonic improvement theory) 
[34]  Algorithms for Reinforcement Learning, Csaba Szepesvari, 2009 (Classic RL Algorithms) 
[35]  Benchmarking Deep Reinforcement Learning for Continuous Control, Duan et al, 2016 
[36]  Playing Atari with Deep Reinforcement Learning, Mnih et al, 2013 (DQN) 
[37]  OpenAI Baselines: ACKTR & A2C 
[38]  Asynchronous Methods for Deep Reinforcement Learning, Mnih et al, 2016 (A3C) 
[39]  Proximal Policy Optimization Algorithms, Schulman et al, 2017 (PPO) 
[40]  Continuous Control with Deep Reinforcement Learning, Lillicrap et al, 2015 (DDPG) 
[41]  RLIntro Policy Gradient Sample Code, Josh Achiam, 2018 
[42]  OpenAI Baselines 
[43]  rllab 
[44]  OpenAI Gym 
[45]  OpenAI Retro Contest 
[46]  OpenAI Gym Retro 
[47]  Center for Open Science, explaining what preregistration means in the context of scientific experiments. 
Key Papers in Deep RLÂ¶
What follows is a list of papers in deep RL that are worth reading. This is far from comprehensive, but should provide a useful starting point for someone looking to do research in the field.
Table of Contents
 Key Papers in Deep RL
 1. ModelFree RL
 2. Exploration
 3. Transfer and Multitask RL
 4. Hierarchy
 5. Memory
 6. ModelBased RL
 7. MetaRL
 8. Scaling RL
 9. RL in the Real World
 10. Safety
 11. Imitation Learning and Inverse Reinforcement Learning
 12. Reproducibility, Analysis, and Critique
 13. Bonus: Classic Papers in RL Theory or Review
1. ModelFree RLÂ¶
a. Deep QLearningÂ¶
[1]  Playing Atari with Deep Reinforcement Learning, Mnih et al, 2013. Algorithm: DQN. 
[2]  Deep Recurrent QLearning for Partially Observable MDPs, Hausknecht and Stone, 2015. Algorithm: Deep Recurrent QLearning. 
[3]  Dueling Network Architectures for Deep Reinforcement Learning, Wang et al, 2015. Algorithm: Dueling DQN. 
[4]  Deep Reinforcement Learning with Double Qlearning, Hasselt et al 2015. Algorithm: Double DQN. 
[5]  Prioritized Experience Replay, Schaul et al, 2015. Algorithm: Prioritized Experience Replay (PER). 
[6]  Rainbow: Combining Improvements in Deep Reinforcement Learning, Hessel et al, 2017. Algorithm: Rainbow DQN. 
b. Policy GradientsÂ¶
[7]  Asynchronous Methods for Deep Reinforcement Learning, Mnih et al, 2016. Algorithm: A3C. 
[8]  Trust Region Policy Optimization, Schulman et al, 2015. Algorithm: TRPO. 
[9]  HighDimensional Continuous Control Using Generalized Advantage Estimation, Schulman et al, 2015. Algorithm: GAE. 
[10]  Proximal Policy Optimization Algorithms, Schulman et al, 2017. Algorithm: PPOClip, PPOPenalty. 
[11]  Emergence of Locomotion Behaviours in Rich Environments, Heess et al, 2017. Algorithm: PPOPenalty. 
[12]  Scalable trustregion method for deep reinforcement learning using Kroneckerfactored approximation, Wu et al, 2017. Algorithm: ACKTR. 
[13]  Sample Efficient ActorCritic with Experience Replay, Wang et al, 2016. Algorithm: ACER. 
[14]  Soft ActorCritic: OffPolicy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor, Haarnoja et al, 2018. Algorithm: SAC. 
c. Deterministic Policy GradientsÂ¶
[15]  Deterministic Policy Gradient Algorithms, Silver et al, 2014. Algorithm: DPG. 
[16]  Continuous Control With Deep Reinforcement Learning, Lillicrap et al, 2015. Algorithm: DDPG. 
[17]  Addressing Function Approximation Error in ActorCritic Methods, Fujimoto et al, 2018. Algorithm: TD3. 
d. Distributional RLÂ¶
[18]  A Distributional Perspective on Reinforcement Learning, Bellemare et al, 2017. Algorithm: C51. 
[19]  Distributional Reinforcement Learning with Quantile Regression, Dabney et al, 2017. Algorithm: QRDQN. 
[20]  Implicit Quantile Networks for Distributional Reinforcement Learning, Dabney et al, 2018. Algorithm: IQN. 
[21]  Dopamine: A Research Framework for Deep Reinforcement Learning, Anonymous, 2018. Contribution: Introduces Dopamine, a code repository containing implementations of DQN, C51, IQN, and Rainbow. Code link. 
e. Policy Gradients with ActionDependent BaselinesÂ¶
[22]  QProp: SampleEfficient Policy Gradient with An OffPolicy Critic, Gu et al, 2016. Algorithm: QProp. 
[23]  Actiondepedent Control Variates for Policy Optimization via Stein’s Identity, Liu et al, 2017. Algorithm: Stein Control Variates. 
[24]  The Mirage of ActionDependent Baselines in Reinforcement Learning, Tucker et al, 2018. Contribution: interestingly, critiques and reevaluates claims from earlier papers (including QProp and stein control variates) and finds important methodological errors in them. 
f. PathConsistency LearningÂ¶
[25]  Bridging the Gap Between Value and Policy Based Reinforcement Learning, Nachum et al, 2017. Algorithm: PCL. 
[26]  TrustPCL: An OffPolicy Trust Region Method for Continuous Control, Nachum et al, 2017. Algorithm: TrustPCL. 
g. Other Directions for Combining PolicyLearning and QLearningÂ¶
[27]  Combining Policy Gradient and Qlearning, O’Donoghue et al, 2016. Algorithm: PGQL. 
[28]  The Reactor: A Fast and SampleEfficient ActorCritic Agent for Reinforcement Learning, Gruslys et al, 2017. Algorithm: Reactor. 
[29]  Interpolated Policy Gradient: Merging OnPolicy and OffPolicy Gradient Estimation for Deep Reinforcement Learning, Gu et al, 2017. Algorithm: IPG. 
[30]  Equivalence Between Policy Gradients and Soft QLearning, Schulman et al, 2017. Contribution: Reveals a theoretical link between these two families of RL algorithms. 
h. Evolutionary AlgorithmsÂ¶
[31]  Evolution Strategies as a Scalable Alternative to Reinforcement Learning, Salimans et al, 2017. Algorithm: ES. 
2. ExplorationÂ¶
a. Intrinsic MotivationÂ¶
[32]  VIME: Variational Information Maximizing Exploration, Houthooft et al, 2016. Algorithm: VIME. 
[33]  Unifying CountBased Exploration and Intrinsic Motivation, Bellemare et al, 2016. Algorithm: CTSbased Pseudocounts. 
[34]  CountBased Exploration with Neural Density Models, Ostrovski et al, 2017. Algorithm: PixelCNNbased Pseudocounts. 
[35]  #Exploration: A Study of CountBased Exploration for Deep Reinforcement Learning, Tang et al, 2016. Algorithm: Hashbased Counts. 
[36]  EX2: Exploration with Exemplar Models for Deep Reinforcement Learning, Fu et al, 2017. Algorithm: EX2. 
[37]  Curiositydriven Exploration by Selfsupervised Prediction, Pathak et al, 2017. Algorithm: Intrinsic Curiosity Module (ICM). 
[38]  LargeScale Study of CuriosityDriven Learning, Burda et al, 2018. Contribution: Systematic analysis of how surprisalbased intrinsic motivation performs in a wide variety of environments. 
[39]  Exploration by Random Network Distillation, Burda et al, 2018. Algorithm: RND. 
b. Unsupervised RLÂ¶
[40]  Variational Intrinsic Control, Gregor et al, 2016. Algorithm: VIC. 
[41]  Diversity is All You Need: Learning Skills without a Reward Function, Eysenbach et al, 2018. Algorithm: DIAYN. 
[42]  Variational Option Discovery Algorithms, Achiam et al, 2018. Algorithm: VALOR. 
3. Transfer and Multitask RLÂ¶
[43]  Progressive Neural Networks, Rusu et al, 2016. Algorithm: Progressive Networks. 
[44]  Universal Value Function Approximators, Schaul et al, 2015. Algorithm: UVFA. 
[45]  Reinforcement Learning with Unsupervised Auxiliary Tasks, Jaderberg et al, 2016. Algorithm: UNREAL. 
[46]  The Intentional Unintentional Agent: Learning to Solve Many Continuous Control Tasks Simultaneously, Cabi et al, 2017. Algorithm: IU Agent. 
[47]  PathNet: Evolution Channels Gradient Descent in Super Neural Networks, Fernando et al, 2017. Algorithm: PathNet. 
[48]  Mutual Alignment Transfer Learning, Wulfmeier et al, 2017. Algorithm: MATL. 
[49]  Learning an Embedding Space for Transferable Robot Skills, Hausman et al, 2018. 
[50]  Hindsight Experience Replay, Andrychowicz et al, 2017. Algorithm: Hindsight Experience Replay (HER). 
4. HierarchyÂ¶
[51]  Strategic Attentive Writer for Learning MacroActions, Vezhnevets et al, 2016. Algorithm: STRAW. 
[52]  FeUdal Networks for Hierarchical Reinforcement Learning, Vezhnevets et al, 2017. Algorithm: Feudal Networks 
[53]  DataEfficient Hierarchical Reinforcement Learning, Nachum et al, 2018. Algorithm: HIRO. 
5. MemoryÂ¶
[54]  ModelFree Episodic Control, Blundell et al, 2016. Algorithm: MFEC. 
[55]  Neural Episodic Control, Pritzel et al, 2017. Algorithm: NEC. 
[56]  Neural Map: Structured Memory for Deep Reinforcement Learning, Parisotto and Salakhutdinov, 2017. Algorithm: Neural Map. 
[57]  Unsupervised Predictive Memory in a GoalDirected Agent, Wayne et al, 2018. Algorithm: MERLIN. 
[58]  Relational Recurrent Neural Networks, Santoro et al, 2018. Algorithm: RMC. 
6. ModelBased RLÂ¶
a. Model is LearnedÂ¶
[59]  ImaginationAugmented Agents for Deep Reinforcement Learning, Weber et al, 2017. Algorithm: I2A. 
[60]  Neural Network Dynamics for ModelBased Deep Reinforcement Learning with ModelFree FineTuning, Nagabandi et al, 2017. Algorithm: MBMF. 
[61]  ModelBased Value Expansion for Efficient ModelFree Reinforcement Learning, Feinberg et al, 2018. Algorithm: MVE. 
[62]  SampleEfficient Reinforcement Learning with Stochastic Ensemble Value Expansion, Buckman et al, 2018. Algorithm: STEVE. 
[63]  ModelEnsemble TrustRegion Policy Optimization, Kurutach et al, 2018. Algorithm: METRPO. 
[64]  ModelBased Reinforcement Learning via MetaPolicy Optimization, Clavera et al, 2018. Algorithm: MBMPO. 
[65]  Recurrent World Models Facilitate Policy Evolution, Ha and Schmidhuber, 2018. 
b. Model is GivenÂ¶
[66]  Mastering Chess and Shogi by SelfPlay with a General Reinforcement Learning Algorithm, Silver et al, 2017. Algorithm: AlphaZero. 
[67]  Thinking Fast and Slow with Deep Learning and Tree Search, Anthony et al, 2017. Algorithm: ExIt. 
7. MetaRLÂ¶
[68]  RL^2: Fast Reinforcement Learning via Slow Reinforcement Learning, Duan et al, 2016. Algorithm: RL^2. 
[69]  Learning to Reinforcement Learn, Wang et al, 2016. 
[70]  ModelAgnostic MetaLearning for Fast Adaptation of Deep Networks, Finn et al, 2017. Algorithm: MAML. 
[71]  A Simple Neural Attentive MetaLearner, Mishra et al, 2018. Algorithm: SNAIL. 
8. Scaling RLÂ¶
[72]  Accelerated Methods for Deep Reinforcement Learning, Stooke and Abbeel, 2018. Contribution: Systematic analysis of parallelization in deep RL across algorithms. 
[73]  IMPALA: Scalable Distributed DeepRL with Importance Weighted ActorLearner Architectures, Espeholt et al, 2018. Algorithm: IMPALA. 
[74]  Distributed Prioritized Experience Replay, Horgan et al, 2018. Algorithm: ApeX. 
[75]  Recurrent Experience Replay in Distributed Reinforcement Learning, Anonymous, 2018. Algorithm: R2D2. 
[76]  RLlib: Abstractions for Distributed Reinforcement Learning, Liang et al, 2017. Contribution: A scalable library of RL algorithm implementations. Documentation link. 
9. RL in the Real WorldÂ¶
[77]  Benchmarking Reinforcement Learning Algorithms on RealWorld Robots, Mahmood et al, 2018. 
[78]  Learning Dexterous InHand Manipulation, OpenAI, 2018. 
[79]  QTOpt: Scalable Deep Reinforcement Learning for VisionBased Robotic Manipulation, Kalashnikov et al, 2018. Algorithm: QTOpt. 
[80]  Horizon: Facebook’s Open Source Applied Reinforcement Learning Platform, Gauci et al, 2018. 
10. SafetyÂ¶
[81]  Concrete Problems in AI Safety, Amodei et al, 2016. Contribution: establishes a taxonomy of safety problems, serving as an important jumpingoff point for future research. We need to solve these! 
[82]  Deep Reinforcement Learning From Human Preferences, Christiano et al, 2017. Algorithm: LFP. 
[83]  Constrained Policy Optimization, Achiam et al, 2017. Algorithm: CPO. 
[84]  Safe Exploration in Continuous Action Spaces, Dalal et al, 2018. Algorithm: DDPG+Safety Layer. 
[85]  Trial without Error: Towards Safe Reinforcement Learning via Human Intervention, Saunders et al, 2017. Algorithm: HIRL. 
[86]  Leave No Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning, Eysenbach et al, 2017. Algorithm: Leave No Trace. 
11. Imitation Learning and Inverse Reinforcement LearningÂ¶
[87]  Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy, Ziebart 2010. Contributions: Crisp formulation of maximum entropy IRL. 
[88]  Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization, Finn et al, 2016. Algorithm: GCL. 
[89]  Generative Adversarial Imitation Learning, Ho and Ermon, 2016. Algorithm: GAIL. 
[90]  DeepMimic: ExampleGuided Deep Reinforcement Learning of PhysicsBased Character Skills, Peng et al, 2018. Algorithm: DeepMimic. 
[91]  Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow, Peng et al, 2018. Algorithm: VAIL. 
[92]  OneShot HighFidelity Imitation: Training LargeScale Deep Nets with RL, Le Paine et al, 2018. Algorithm: MetaMimic. 
12. Reproducibility, Analysis, and CritiqueÂ¶
[93]  Benchmarking Deep Reinforcement Learning for Continuous Control, Duan et al, 2016. Contribution: rllab. 
[94]  Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control, Islam et al, 2017. 
[95]  Deep Reinforcement Learning that Matters, Henderson et al, 2017. 
[96]  Where Did My Optimum Go?: An Empirical Analysis of Gradient Descent Optimization in Policy Gradient Methods, Henderson et al, 2018. 
[97]  Are Deep Policy Gradient Algorithms Truly Policy Gradient Algorithms?, Ilyas et al, 2018. 
[98]  Simple Random Search Provides a Competitive Approach to Reinforcement Learning, Mania et al, 2018. 
[99]  Benchmarking ModelBased Reinforcement Learning, Wang et al, 2019. 
13. Bonus: Classic Papers in RL Theory or ReviewÂ¶
[100]  Policy Gradient Methods for Reinforcement Learning with Function Approximation, Sutton et al, 2000. Contributions: Established policy gradient theorem and showed convergence of policy gradient algorithm for arbitrary policy classes. 
[101]  An Analysis of TemporalDifference Learning with Function Approximation, Tsitsiklis and Van Roy, 1997. Contributions: Variety of convergence results and counterexamples for valuelearning methods in RL. 
[102]  Reinforcement Learning of Motor Skills with Policy Gradients, Peters and Schaal, 2008. Contributions: Thorough review of policy gradient methods at the time, many of which are still serviceable descriptions of deep RL methods. 
[103]  Approximately Optimal Approximate Reinforcement Learning, Kakade and Langford, 2002. Contributions: Early roots for monotonic improvement theory, later leading to theoretical justification for TRPO and other algorithms. 
[104]  A Natural Policy Gradient, Kakade, 2002. Contributions: Brought natural gradients into RL, later leading to TRPO, ACKTR, and several other methods in deep RL. 
[105]  Algorithms for Reinforcement Learning, Szepesvari, 2009. Contributions: Unbeatable reference on RL before deep RL, containing foundations and theoretical background. 
ExercisesÂ¶
Table of Contents
Problem Set 1: Basics of ImplementationÂ¶
Exercise 1.1: Gaussian LogLikelihood
Path to Exercise:
 PyTorch version:
spinup/exercises/pytorch/problem_set_1/exercise1_1.py
 Tensorflow version:
spinup/exercises/tf1/problem_set_1/exercise1_1.py
Path to Solution:
 PyTorch version:
spinup/exercises/pytorch/problem_set_1_solutions/exercise1_1_soln.py
 Tensorflow version:
spinup/exercises/tf1/problem_set_1_solutions/exercise1_1_soln.py
Instructions. Write a function that takes in the means and log stds of a batch of diagonal Gaussian distributions, along with (previouslygenerated) samples from those distributions, and returns the log likelihoods of those samples. (In the Tensorflow version, you will write a function that creates computation graph operations to do this; in the PyTorch version, you will directly operate on given Tensors.)
You may find it useful to review the formula given in this section of the RL introduction.
Implement your solution in exercise1_1.py
, and run that file to automatically check your work.
Evaluation Criteria. Your solution will be checked by comparing outputs against a knowngood implementation, using a batch of random inputs.
Exercise 1.2: Policy for PPO
Path to Exercise:
 PyTorch version:
spinup/exercises/pytorch/problem_set_1/exercise1_2.py
 Tensorflow version:
spinup/exercises/tf1/problem_set_1/exercise1_2.py
Path to Solution:
 PyTorch version:
spinup/exercises/pytorch/problem_set_1_solutions/exercise1_2_soln.py
 Tensorflow version:
spinup/exercises/tf1/problem_set_1_solutions/exercise1_2_soln.py
Instructions. Implement an MLP diagonal Gaussian policy for PPO.
Implement your solution in exercise1_2.py
, and run that file to automatically check your work.
Evaluation Criteria. Your solution will be evaluated by running for 20 epochs in the InvertedPendulumv2 Gym environment, and this should take in the ballpark of 35 minutes (depending on your machine, and other processes you are running in the background). The bar for success is reaching an average score of over 500 in the last 5 epochs, or getting to a score of 1000 (the maximum possible score) in the last 5 epochs.
Exercise 1.3: Computation Graph for TD3
Path to Exercise.
 PyTorch version:
spinup/exercises/pytorch/problem_set_1/exercise1_3.py
 Tensorflow version:
spinup/exercises/tf1/problem_set_1/exercise1_3.py
Path to Solution.
 PyTorch version:
spinup/algos/pytorch/td3/td3.py
 Tensorflow version:
spinup/algos/tf1/td3/td3.py
Instructions. Implement the main mathematical logic for the TD3 algorithm.
As starter code, you are given the entirety of the TD3 algorithm except for the main mathematical logic (essentially, the loss functions and intermediate calculations needed for them). Find “YOUR CODE HERE” to begin.
You may find it useful to review the pseudocode in our page on TD3.
Implement your solution in exercise1_3.py
, and run that file to see the results of your work. There is no automatic checking for this exercise.
Evaluation Criteria. Evaluate your code by running exercise1_3.py
with HalfCheetahv2, InvertedPendulumv2, and one other Gym MuJoCo environment of your choosing (set via the env
flag). It is set up to use smaller neural networks (hidden sizes [128,128]) than typical for TD3, with a maximum episode length of 150, and to run for only 10 epochs. The goal is to see significant learning progress relatively quickly (in terms of wall clock time). Experiments will likely take on the order of ~10 minutes.
Use the use_soln
flag to run Spinning Up’s TD3 instead of your implementation. Anecdotally, within 10 epochs, the score in HalfCheetah should go over 300, and the score in InvertedPendulum should max out at 150.
Problem Set 2: Algorithm Failure ModesÂ¶
Exercise 2.1: Value Function Fitting in TRPO
Path to Exercise. (Not applicable, there is no code for this one.)
Path to Solution. Solution available here.
Many factors can impact the performance of policy gradient algorithms, but few more drastically than the quality of the learned value function used for advantage estimation.
In this exercise, you will compare results between runs of TRPO where you put lots of effort into fitting the value function (train_v_iters=80
), versus where you put very little effort into fitting the value function (train_v_iters=0
).
Instructions. Run the following command:
python m spinup.run trpo env Hopperv2 train_v_iters[v] 0 80 exp_name ex21 epochs 250 steps_per_epoch 4000 seed 0 10 20 dt
and plot the results. (These experiments might take ~10 minutes each, and this command runs six of them.) What do you find?
Exercise 2.2: Silent Bug in DDPG
Path to Exercise.
 PyTorch version:
spinup/exercises/pytorch/problem_set_2/exercise2_2.py
 Tensorflow version:
spinup/exercises/tf1/problem_set_2/exercise2_2.py
Path to Solution. Solution available here.
The hardest part of writing RL code is dealing with bugs, because failures are frequently silent. The code will appear to run correctly, but the agent’s performance will degrade relative to a bugfree implementation—sometimes to the extent that it never learns anything.
In this exercise, you will observe a bug in vivo and compare results against correct code. The bug is the same (conceptually, if not in exact implementation) for both the PyTorch and Tensorflow versions of this exercise.
Instructions. Run exercise2_2.py
, which will launch DDPG experiments with and without a bug. The nonbugged version runs the default Spinning Up implementation of DDPG, using a default method for creating the actor and critic networks. The bugged version runs the same DDPG code, except uses a bugged method for creating the networks.
There will be six experiments in all (three random seeds for each case), and each should take in the ballpark of 10 minutes. When they’re finished, plot the results. What is the difference in performance with and without the bug?
Without referencing the correct actorcritic code (which is to say—don’t look in DDPG’s core.py
file), try to figure out what the bug is and explain how it breaks things.
Hint. To figure out what’s going wrong, think about how the DDPG code implements the DDPG computation graph. For the Tensorflow version, look at this excerpt:
# Bellman backup for Q function
backup = tf.stop_gradient(r_ph + gamma*(1d_ph)*q_pi_targ)
# DDPG losses
pi_loss = tf.reduce_mean(q_pi)
q_loss = tf.reduce_mean((qbackup)**2)
How could a bug in the actorcritic code have an impact here?
Bonus. Are there any choices of hyperparameters which would have hidden the effects of the bug?
ChallengesÂ¶
Write Code from Scratch
As we suggest in the essay, try reimplementing various deep RL algorithms from scratch.
Requests for Research
If you feel comfortable with writing deep learning and deep RL code, consider trying to make progress on any of OpenAI’s standing requests for research:
Benchmarks for Spinning Up ImplementationsÂ¶
Table of Contents
We benchmarked the Spinning Up algorithm implementations in five environments from the MuJoCo Gym task suite: HalfCheetah, Hopper, Walker2d, Swimmer, and Ant.
Experiment DetailsÂ¶
Random seeds. All experiments were run for 10 random seeds each. Graphs show the average (solid line) and std dev (shaded) of performance over random seed over the course of training.
Performance metric. Performance for the onpolicy algorithms is measured as the average trajectory return across the batch collected at each epoch. Performance for the offpolicy algorithms is measured once every 10,000 steps by running the deterministic policy (or, in the case of SAC, the mean policy) without action noise for ten trajectories, and reporting the average return over those test trajectories.
Network architectures. The onpolicy algorithms use networks of size (64, 32) with tanh units for both the policy and the value function. The offpolicy algorithms use networks of size (256, 256) with relu units.
Batch size. The onpolicy algorithms collected 4000 steps of agentenvironment interaction per batch update. The offpolicy algorithms used minibatches of size 100 at each gradient descent step.
All other hyperparameters are left at default settings for the Spinning Up implementations. See algorithm pages for details.
Learning curves are smoothed by averaging over a window of 11 epochs.
You Should Know
By comparison to the literature, the Spinning Up implementations of DDPG, TD3, and SAC are roughly atparity with the best reported results for these algorithms. As a result, you can use the Spinning Up implementations of these algorithms for research purposes.
The Spinning Up implementations of VPG, TRPO, and PPO are overall a bit weaker than the best reported results for these algorithms. This is due to the absence of some standard tricks (such as observation normalization and normalized value regression targets) from our implementations. For research comparisons, you should use the implementations of TRPO or PPO from OpenAI Baselines.
PyTorch vs TensorflowÂ¶
We provide graphs for headtohead comparisons between the PyTorch and Tensorflow implementations of each algorithm at the following pages:
Vanilla Policy GradientÂ¶
Table of Contents
BackgroundÂ¶
(Previously: Introduction to RL, Part 3)
The key idea underlying policy gradients is to push up the probabilities of actions that lead to higher return, and push down the probabilities of actions that lead to lower return, until you arrive at the optimal policy.
Quick FactsÂ¶
 VPG is an onpolicy algorithm.
 VPG can be used for environments with either discrete or continuous action spaces.
 The Spinning Up implementation of VPG supports parallelization with MPI.
Key EquationsÂ¶
Let denote a policy with parameters , and denote the expected finitehorizon undiscounted return of the policy. The gradient of is
where is a trajectory and is the advantage function for the current policy.
The policy gradient algorithm works by updating policy parameters via stochastic gradient ascent on policy performance:
Policy gradient implementations typically compute advantage function estimates based on the infinitehorizon discounted return, despite otherwise using the finitehorizon undiscounted policy gradient formula.
Exploration vs. ExploitationÂ¶
VPG trains a stochastic policy in an onpolicy way. This means that it explores by sampling actions according to the latest version of its stochastic policy. The amount of randomness in action selection depends on both initial conditions and the training procedure. Over the course of training, the policy typically becomes progressively less random, as the update rule encourages it to exploit rewards that it has already found. This may cause the policy to get trapped in local optima.
DocumentationÂ¶
You Should Know
In what follows, we give documentation for the PyTorch and Tensorflow implementations of VPG in Spinning Up. They have nearly identical function calls and docstrings, except for details relating to model construction. However, we include both full docstrings for completeness.
Documentation: PyTorch VersionÂ¶

spinup.
vpg_pytorch
(env_fn, actor_critic=<MagicMock spec='str' id='140191853362816'>, ac_kwargs={}, seed=0, steps_per_epoch=4000, epochs=50, gamma=0.99, pi_lr=0.0003, vf_lr=0.001, train_v_iters=80, lam=0.97, max_ep_len=1000, logger_kwargs={}, save_freq=10)Â¶ Vanilla Policy Gradient
(with GAELambda for advantage estimation)
Parameters:  env_fn – A function which creates a copy of the environment. The environment must satisfy the OpenAI Gym API.
 actor_critic –
The constructor method for a PyTorch Module with a
step
method, anact
method, api
module, and av
module. Thestep
method should accept a batch of observations and return:Symbol Shape Description a
(batch, act_dim) Numpy array of actions for eachobservation.v
(batch,) Numpy array of value estimatesfor the provided observations.logp_a
(batch,) Numpy array of log probs for theactions ina
.The
act
method behaves the same asstep
but only returnsa
.The
pi
module’s forward call should accept a batch of observations and optionally a batch of actions, and return:Symbol Shape Description pi
N/A Torch Distribution object, containinga batch of distributions describingthe policy for the provided observations.logp_a
(batch,) Optional (only returned if batch ofactions is given). Tensor containingthe log probability, according tothe policy, of the provided actions.If actions not given, will containNone
.The
v
module’s forward call should accept a batch of observations and return:Symbol Shape Description v
(batch,) Tensor containing the value estimatesfor the provided observations. (Critical:make sure to flatten this!)  ac_kwargs (dict) – Any kwargs appropriate for the ActorCritic object you provided to VPG.
 seed (int) – Seed for random number generators.
 steps_per_epoch (int) – Number of steps of interaction (stateaction pairs) for the agent and the environment in each epoch.
 epochs (int) – Number of epochs of interaction (equivalent to number of policy updates) to perform.
 gamma (float) – Discount factor. (Always between 0 and 1.)
 pi_lr (float) – Learning rate for policy optimizer.
 vf_lr (float) – Learning rate for value function optimizer.
 train_v_iters (int) – Number of gradient descent steps to take on value function per epoch.
 lam (float) – Lambda for GAELambda. (Always between 0 and 1, close to 1.)
 max_ep_len (int) – Maximum length of trajectory / episode / rollout.
 logger_kwargs (dict) – Keyword args for EpochLogger.
 save_freq (int) – How often (in terms of gap between epochs) to save the current policy and value function.
Saved Model Contents: PyTorch VersionÂ¶
The PyTorch saved model can be loaded with ac = torch.load('path/to/model.pt')
, yielding an actorcritic object (ac
) that has the properties described in the docstring for vpg_pytorch
.
You can get actions from this model with
actions = ac.act(torch.as_tensor(obs, dtype=torch.float32))
Documentation: Tensorflow VersionÂ¶

spinup.
vpg_tf1
(env_fn, actor_critic=<function mlp_actor_critic>, ac_kwargs={}, seed=0, steps_per_epoch=4000, epochs=50, gamma=0.99, pi_lr=0.0003, vf_lr=0.001, train_v_iters=80, lam=0.97, max_ep_len=1000, logger_kwargs={}, save_freq=10)Â¶ Vanilla Policy Gradient
(with GAELambda for advantage estimation)
Parameters:  env_fn – A function which creates a copy of the environment. The environment must satisfy the OpenAI Gym API.
 actor_critic –
A function which takes in placeholder symbols for state,
x_ph
, and action,a_ph
, and returns the main outputs from the agent’s Tensorflow computation graph:Symbol Shape Description pi
(batch, act_dim) Samples actions from policy givenstates.logp
(batch,) Gives log probability, according tothe policy, of taking actionsa_ph
in statesx_ph
.logp_pi
(batch,) Gives log probability, according tothe policy, of the action sampled bypi
.v
(batch,) Gives the value estimate for statesinx_ph
. (Critical: make sureto flatten this!)  ac_kwargs (dict) – Any kwargs appropriate for the actor_critic function you provided to VPG.
 seed (int) – Seed for random number generators.
 steps_per_epoch (int) – Number of steps of interaction (stateaction pairs) for the agent and the environment in each epoch.
 epochs (int) – Number of epochs of interaction (equivalent to number of policy updates) to perform.
 gamma (float) – Discount factor. (Always between 0 and 1.)
 pi_lr (float) – Learning rate for policy optimizer.
 vf_lr (float) – Learning rate for value function optimizer.
 train_v_iters (int) – Number of gradient descent steps to take on value function per epoch.
 lam (float) – Lambda for GAELambda. (Always between 0 and 1, close to 1.)
 max_ep_len (int) – Maximum length of trajectory / episode / rollout.
 logger_kwargs (dict) – Keyword args for EpochLogger.
 save_freq (int) – How often (in terms of gap between epochs) to save the current policy and value function.
Saved Model Contents: Tensorflow VersionÂ¶
The computation graph saved by the logger includes:
Key  Value 

x 
Tensorflow placeholder for state input. 
pi 
Samples an action from the agent, conditioned on states in x . 
v 
Gives value estimate for states in x . 
This saved model can be accessed either by
 running the trained policy with the test_policy.py tool,
 or loading the whole saved graph into a program with restore_tf_graph.
ReferencesÂ¶
Relevant PapersÂ¶
 Policy Gradient Methods for Reinforcement Learning with Function Approximation, Sutton et al. 2000
 Optimizing Expectations: From Deep Reinforcement Learning to Stochastic Computation Graphs, Schulman 2016(a)
 Benchmarking Deep Reinforcement Learning for Continuous Control, Duan et al. 2016
 High Dimensional Continuous Control Using Generalized Advantage Estimation, Schulman et al. 2016(b)
Why These Papers?Â¶
Sutton 2000 is included because it is a timeless classic of reinforcement learning theory, and contains references to the earlier work which led to modern policy gradients. Schulman 2016(a) is included because Chapter 2 contains a lucid introduction to the theory of policy gradient algorithms, including pseudocode. Duan 2016 is a clear, recent benchmark paper that shows how vanilla policy gradient in the deep RL setting (eg with neural network policies and Adam as the optimizer) compares with other deep RL algorithms. Schulman 2016(b) is included because our implementation of VPG makes use of Generalized Advantage Estimation for computing the policy gradient.
Trust Region Policy OptimizationÂ¶
Table of Contents
BackgroundÂ¶
(Previously: Background for VPG)
TRPO updates policies by taking the largest step possible to improve performance, while satisfying a special constraint on how close the new and old policies are allowed to be. The constraint is expressed in terms of KLDivergence, a measure of (something like, but not exactly) distance between probability distributions.
This is different from normal policy gradient, which keeps new and old policies close in parameter space. But even seemingly small differences in parameter space can have very large differences in performance—so a single bad step can collapse the policy performance. This makes it dangerous to use large step sizes with vanilla policy gradients, thus hurting its sample efficiency. TRPO nicely avoids this kind of collapse, and tends to quickly and monotonically improve performance.
Quick FactsÂ¶
 TRPO is an onpolicy algorithm.
 TRPO can be used for environments with either discrete or continuous action spaces.
 The Spinning Up implementation of TRPO supports parallelization with MPI.
Key EquationsÂ¶
Let denote a policy with parameters . The theoretical TRPO update is:
where is the surrogate advantage, a measure of how policy performs relative to the old policy using data from the old policy:
and is an average KLdivergence between policies across states visited by the old policy:
You Should Know
The objective and constraint are both zero when . Furthermore, the gradient of the constraint with respect to is zero when . Proving these facts requires some subtle command of the relevant math—it’s an exercise worth doing, whenever you feel ready!
The theoretical TRPO update isn’t the easiest to work with, so TRPO makes some approximations to get an answer quickly. We Taylor expand the objective and constraint to leading order around :
resulting in an approximate optimization problem,
You Should Know
By happy coincidence, the gradient of the surrogate advantage function with respect to , evaluated at , is exactly equal to the policy gradient, ! Try proving this, if you feel comfortable diving into the math.
This approximate problem can be analytically solved by the methods of Lagrangian duality [1], yielding the solution:
If we were to stop here, and just use this final result, the algorithm would be exactly calculating the Natural Policy Gradient. A problem is that, due to the approximation errors introduced by the Taylor expansion, this may not satisfy the KL constraint, or actually improve the surrogate advantage. TRPO adds a modification to this update rule: a backtracking line search,
where is the backtracking coefficient, and is the smallest nonnegative integer such that satisfies the KL constraint and produces a positive surrogate advantage.
Lastly: computing and storing the matrix inverse, , is painfully expensive when dealing with neural network policies with thousands or millions of parameters. TRPO sidesteps the issue by using the conjugate gradient algorithm to solve for , requiring only a function which can compute the matrixvector product instead of computing and storing the whole matrix directly. This is not too hard to do: we set up a symbolic operation to calculate
which gives us the correct output without computing the whole matrix.
[1]  See Convex Optimization by Boyd and Vandenberghe, especially chapters 2 through 5. 
Exploration vs. ExploitationÂ¶
TRPO trains a stochastic policy in an onpolicy way. This means that it explores by sampling actions according to the latest version of its stochastic policy. The amount of randomness in action selection depends on both initial conditions and the training procedure. Over the course of training, the policy typically becomes progressively less random, as the update rule encourages it to exploit rewards that it has already found. This may cause the policy to get trapped in local optima.
DocumentationÂ¶
You Should Know
Spinning Up currently only has a Tensorflow implementation of TRPO.

spinup.
trpo_tf1
(env_fn, actor_critic=<function mlp_actor_critic>, ac_kwargs={}, seed=0, steps_per_epoch=4000, epochs=50, gamma=0.99, delta=0.01, vf_lr=0.001, train_v_iters=80, damping_coeff=0.1, cg_iters=10, backtrack_iters=10, backtrack_coeff=0.8, lam=0.97, max_ep_len=1000, logger_kwargs={}, save_freq=10, algo='trpo')Â¶ Trust Region Policy Optimization
(with support for Natural Policy Gradient)
Parameters:  env_fn – A function which creates a copy of the environment. The environment must satisfy the OpenAI Gym API.
 actor_critic –
A function which takes in placeholder symbols for state,
x_ph
, and action,a_ph
, and returns the main outputs from the agent’s Tensorflow computation graph:Symbol Shape Description pi
(batch, act_dim) Samples actions from policy givenstates.logp
(batch,) Gives log probability, according tothe policy, of taking actionsa_ph
in statesx_ph
.logp_pi
(batch,) Gives log probability, according tothe policy, of the action sampled bypi
.info
N/A A dict of any intermediate quantities(from calculating the policy or logprobabilities) which are needed foranalytically computing KL divergence.(eg sufficient statistics of thedistributions)info_phs
N/A A dict of placeholders for old valuesof the entries ininfo
.d_kl
() A symbol for computing the mean KLdivergence between the current policy(pi
) and the old policy (asspecified by the inputs toinfo_phs
) over the batch ofstates given inx_ph
.v
(batch,) Gives the value estimate for statesinx_ph
. (Critical: make sureto flatten this!)  ac_kwargs (dict) – Any kwargs appropriate for the actor_critic function you provided to TRPO.
 seed (int) – Seed for random number generators.
 steps_per_epoch (int) – Number of steps of interaction (stateaction pairs) for the agent and the environment in each epoch.
 epochs (int) – Number of epochs of interaction (equivalent to number of policy updates) to perform.
 gamma (float) – Discount factor. (Always between 0 and 1.)
 delta (float) – KLdivergence limit for TRPO / NPG update. (Should be small for stability. Values like 0.01, 0.05.)
 vf_lr (float) – Learning rate for value function optimizer.
 train_v_iters (int) – Number of gradient descent steps to take on value function per epoch.
 damping_coeff (float) –
Artifact for numerical stability, should be smallish. Adjusts Hessianvector product calculation:
where is the damping coefficient. Probably don’t play with this hyperparameter.
 cg_iters (int) –
Number of iterations of conjugate gradient to perform. Increasing this will lead to a more accurate approximation to , and possibly slightlyimproved performance, but at the cost of slowing things down.
Also probably don’t play with this hyperparameter.
 backtrack_iters (int) – Maximum number of steps allowed in the backtracking line search. Since the line search usually doesn’t backtrack, and usually only steps back once when it does, this hyperparameter doesn’t often matter.
 backtrack_coeff (float) – How far back to step during backtracking line search. (Always between 0 and 1, usually above 0.5.)
 lam (float) – Lambda for GAELambda. (Always between 0 and 1, close to 1.)
 max_ep_len (int) – Maximum length of trajectory / episode / rollout.
 logger_kwargs (dict) – Keyword args for EpochLogger.
 save_freq (int) – How often (in terms of gap between epochs) to save the current policy and value function.
 algo – Either ‘trpo’ or ‘npg’: this code supports both, since they are almost the same.
Saved Model ContentsÂ¶
The computation graph saved by the logger includes:
Key  Value 

x 
Tensorflow placeholder for state input. 
pi 
Samples an action from the agent, conditioned on states in x . 
v 
Gives value estimate for states in x . 
This saved model can be accessed either by
 running the trained policy with the test_policy.py tool,
 or loading the whole saved graph into a program with restore_tf_graph.
ReferencesÂ¶
Relevant PapersÂ¶
 Trust Region Policy Optimization, Schulman et al. 2015
 High Dimensional Continuous Control Using Generalized Advantage Estimation, Schulman et al. 2016
 Approximately Optimal Approximate Reinforcement Learning, Kakade and Langford 2002
Why These Papers?Â¶
Schulman 2015 is included because it is the original paper describing TRPO. Schulman 2016 is included because our implementation of TRPO makes use of Generalized Advantage Estimation for computing the policy gradient. Kakade and Langford 2002 is included because it contains theoretical results which motivate and deeply connect to the theoretical foundations of TRPO.
Proximal Policy OptimizationÂ¶
Table of Contents
BackgroundÂ¶
(Previously: Background for TRPO)
PPO is motivated by the same question as TRPO: how can we take the biggest possible improvement step on a policy using the data we currently have, without stepping so far that we accidentally cause performance collapse? Where TRPO tries to solve this problem with a complex secondorder method, PPO is a family of firstorder methods that use a few other tricks to keep new policies close to old. PPO methods are significantly simpler to implement, and empirically seem to perform at least as well as TRPO.
There are two primary variants of PPO: PPOPenalty and PPOClip.
PPOPenalty approximately solves a KLconstrained update like TRPO, but penalizes the KLdivergence in the objective function instead of making it a hard constraint, and automatically adjusts the penalty coefficient over the course of training so that it’s scaled appropriately.
PPOClip doesn’t have a KLdivergence term in the objective and doesn’t have a constraint at all. Instead relies on specialized clipping in the objective function to remove incentives for the new policy to get far from the old policy.
Here, we’ll focus only on PPOClip (the primary variant used at OpenAI).
Quick FactsÂ¶
 PPO is an onpolicy algorithm.
 PPO can be used for environments with either discrete or continuous action spaces.
 The Spinning Up implementation of PPO supports parallelization with MPI.
Key EquationsÂ¶
PPOclip updates policies via
typically taking multiple steps of (usually minibatch) SGD to maximize the objective. Here is given by
in which is a (small) hyperparameter which roughly says how far away the new policy is allowed to go from the old.
This is a pretty complex expression, and it’s hard to tell at first glance what it’s doing, or how it helps keep the new policy close to the old policy. As it turns out, there’s a considerably simplified version [1] of this objective which is a bit easier to grapple with (and is also the version we implement in our code):
where
To figure out what intuition to take away from this, let’s look at a single stateaction pair , and think of cases.
Advantage is positive: Suppose the advantage for that stateaction pair is positive, in which case its contribution to the objective reduces to
Because the advantage is positive, the objective will increase if the action becomes more likely—that is, if increases. But the min in this term puts a limit to how much the objective can increase. Once , the min kicks in and this term hits a ceiling of . Thus: the new policy does not benefit by going far away from the old policy.
Advantage is negative: Suppose the advantage for that stateaction pair is negative, in which case its contribution to the objective reduces to
Because the advantage is negative, the objective will increase if the action becomes less likely—that is, if decreases. But the max in this term puts a limit to how much the objective can increase. Once , the max kicks in and this term hits a ceiling of . Thus, again: the new policy does not benefit by going far away from the old policy.
What we have seen so far is that clipping serves as a regularizer by removing incentives for the policy to change dramatically, and the hyperparameter corresponds to how far away the new policy can go from the old while still profiting the objective.
You Should Know
While this kind of clipping goes a long way towards ensuring reasonable policy updates, it is still possible to end up with a new policy which is too far from the old policy, and there are a bunch of tricks used by different PPO implementations to stave this off. In our implementation here, we use a particularly simple method: early stopping. If the mean KLdivergence of the new policy from the old grows beyond a threshold, we stop taking gradient steps.
When you feel comfortable with the basic math and implementation details, it’s worth checking out other implementations to see how they handle this issue!
[1]  See this note for a derivation of the simplified form of the PPOClip objective. 
Exploration vs. ExploitationÂ¶
PPO trains a stochastic policy in an onpolicy way. This means that it explores by sampling actions according to the latest version of its stochastic policy. The amount of randomness in action selection depends on both initial conditions and the training procedure. Over the course of training, the policy typically becomes progressively less random, as the update rule encourages it to exploit rewards that it has already found. This may cause the policy to get trapped in local optima.
DocumentationÂ¶
You Should Know
In what follows, we give documentation for the PyTorch and Tensorflow implementations of PPO in Spinning Up. They have nearly identical function calls and docstrings, except for details relating to model construction. However, we include both full docstrings for completeness.
Documentation: PyTorch VersionÂ¶

spinup.
ppo_pytorch
(env_fn, actor_critic=<MagicMock spec='str' id='140191856630472'>, ac_kwargs={}, seed=0, steps_per_epoch=4000, epochs=50, gamma=0.99, clip_ratio=0.2, pi_lr=0.0003, vf_lr=0.001, train_pi_iters=80, train_v_iters=80, lam=0.97, max_ep_len=1000, target_kl=0.01, logger_kwargs={}, save_freq=10)Â¶ Proximal Policy Optimization (by clipping),
with early stopping based on approximate KL
Parameters:  env_fn – A function which creates a copy of the environment. The environment must satisfy the OpenAI Gym API.
 actor_critic –
The constructor method for a PyTorch Module with a
step
method, anact
method, api
module, and av
module. Thestep
method should accept a batch of observations and return:Symbol Shape Description a
(batch, act_dim) Numpy array of actions for eachobservation.v
(batch,) Numpy array of value estimatesfor the provided observations.logp_a
(batch,) Numpy array of log probs for theactions ina
.The
act
method behaves the same asstep
but only returnsa
.The
pi
module’s forward call should accept a batch of observations and optionally a batch of actions, and return:Symbol Shape Description pi
N/A Torch Distribution object, containinga batch of distributions describingthe policy for the provided observations.logp_a
(batch,) Optional (only returned if batch ofactions is given). Tensor containingthe log probability, according tothe policy, of the provided actions.If actions not given, will containNone
.The
v
module’s forward call should accept a batch of observations and return:Symbol Shape Description v
(batch,) Tensor containing the value estimatesfor the provided observations. (Critical:make sure to flatten this!)  ac_kwargs (dict) – Any kwargs appropriate for the ActorCritic object you provided to PPO.
 seed (int) – Seed for random number generators.
 steps_per_epoch (int) – Number of steps of interaction (stateaction pairs) for the agent and the environment in each epoch.
 epochs (int) – Number of epochs of interaction (equivalent to number of policy updates) to perform.
 gamma (float) – Discount factor. (Always between 0 and 1.)
 clip_ratio (float) – Hyperparameter for clipping in the policy objective. Roughly: how far can the new policy go from the old policy while still profiting (improving the objective function)? The new policy can still go farther than the clip_ratio says, but it doesn’t help on the objective anymore. (Usually small, 0.1 to 0.3.) Typically denoted by .
 pi_lr (float) – Learning rate for policy optimizer.
 vf_lr (float) – Learning rate for value function optimizer.
 train_pi_iters (int) – Maximum number of gradient descent steps to take on policy loss per epoch. (Early stopping may cause optimizer to take fewer than this.)
 train_v_iters (int) – Number of gradient descent steps to take on value function per epoch.
 lam (float) – Lambda for GAELambda. (Always between 0 and 1, close to 1.)
 max_ep_len (int) – Maximum length of trajectory / episode / rollout.
 target_kl (float) – Roughly what KL divergence we think is appropriate between new and old policies after an update. This will get used for early stopping. (Usually small, 0.01 or 0.05.)
 logger_kwargs (dict) – Keyword args for EpochLogger.
 save_freq (int) – How often (in terms of gap between epochs) to save the current policy and value function.
Saved Model Contents: PyTorch VersionÂ¶
The PyTorch saved model can be loaded with ac = torch.load('path/to/model.pt')
, yielding an actorcritic object (ac
) that has the properties described in the docstring for ppo_pytorch
.
You can get actions from this model with
actions = ac.act(torch.as_tensor(obs, dtype=torch.float32))
Documentation: Tensorflow VersionÂ¶

spinup.
ppo_tf1
(env_fn, actor_critic=<function mlp_actor_critic>, ac_kwargs={}, seed=0, steps_per_epoch=4000, epochs=50, gamma=0.99, clip_ratio=0.2, pi_lr=0.0003, vf_lr=0.001, train_pi_iters=80, train_v_iters=80, lam=0.97, max_ep_len=1000, target_kl=0.01, logger_kwargs={}, save_freq=10)Â¶ Proximal Policy Optimization (by clipping),
with early stopping based on approximate KL
Parameters:  env_fn – A function which creates a copy of the environment. The environment must satisfy the OpenAI Gym API.
 actor_critic –
A function which takes in placeholder symbols for state,
x_ph
, and action,a_ph
, and returns the main outputs from the agent’s Tensorflow computation graph:Symbol Shape Description pi
(batch, act_dim) Samples actions from policy givenstates.logp
(batch,) Gives log probability, according tothe policy, of taking actionsa_ph
in statesx_ph
.logp_pi
(batch,) Gives log probability, according tothe policy, of the action sampled bypi
.v
(batch,) Gives the value estimate for statesinx_ph
. (Critical: make sureto flatten this!)  ac_kwargs (dict) – Any kwargs appropriate for the actor_critic function you provided to PPO.
 seed (int) – Seed for random number generators.
 steps_per_epoch (int) – Number of steps of interaction (stateaction pairs) for the agent and the environment in each epoch.
 epochs (int) – Number of epochs of interaction (equivalent to number of policy updates) to perform.
 gamma (float) – Discount factor. (Always between 0 and 1.)
 clip_ratio (float) – Hyperparameter for clipping in the policy objective. Roughly: how far can the new policy go from the old policy while still profiting (improving the objective function)? The new policy can still go farther than the clip_ratio says, but it doesn’t help on the objective anymore. (Usually small, 0.1 to 0.3.) Typically denoted by .
 pi_lr (float) – Learning rate for policy optimizer.
 vf_lr (float) – Learning rate for value function optimizer.
 train_pi_iters (int) – Maximum number of gradient descent steps to take on policy loss per epoch. (Early stopping may cause optimizer to take fewer than this.)
 train_v_iters (int) – Number of gradient descent steps to take on value function per epoch.
 lam (float) – Lambda for GAELambda. (Always between 0 and 1, close to 1.)
 max_ep_len (int) – Maximum length of trajectory / episode / rollout.
 target_kl (float) – Roughly what KL divergence we think is appropriate between new and old policies after an update. This will get used for early stopping. (Usually small, 0.01 or 0.05.)
 logger_kwargs (dict) – Keyword args for EpochLogger.
 save_freq (int) – How often (in terms of gap between epochs) to save the current policy and value function.
Saved Model Contents: Tensorflow VersionÂ¶
The computation graph saved by the logger includes:
Key  Value 

x 
Tensorflow placeholder for state input. 
pi 
Samples an action from the agent, conditioned on states in x . 
v 
Gives value estimate for states in x . 
This saved model can be accessed either by
 running the trained policy with the test_policy.py tool,
 or loading the whole saved graph into a program with restore_tf_graph.
ReferencesÂ¶
Relevant PapersÂ¶
 Proximal Policy Optimization Algorithms, Schulman et al. 2017
 High Dimensional Continuous Control Using Generalized Advantage Estimation, Schulman et al. 2016
 Emergence of Locomotion Behaviours in Rich Environments, Heess et al. 2017
Why These Papers?Â¶
Schulman 2017 is included because it is the original paper describing PPO. Schulman 2016 is included because our implementation of PPO makes use of Generalized Advantage Estimation for computing the policy gradient. Heess 2017 is included because it presents a largescale empirical analysis of behaviors learned by PPO agents in complex environments (although it uses PPOpenalty instead of PPOclip).
Other Public ImplementationsÂ¶
 Baselines
 ModularRL (Caution: this implements PPOpenalty instead of PPOclip.)
 rllab (Caution: this implements PPOpenalty instead of PPOclip.)
 rllib (Ray)
Deep Deterministic Policy GradientÂ¶
Table of Contents
BackgroundÂ¶
(Previously: Introduction to RL Part 1: The Optimal QFunction and the Optimal Action)
Deep Deterministic Policy Gradient (DDPG) is an algorithm which concurrently learns a Qfunction and a policy. It uses offpolicy data and the Bellman equation to learn the Qfunction, and uses the Qfunction to learn the policy.
This approach is closely connected to Qlearning, and is motivated the same way: if you know the optimal actionvalue function , then in any given state, the optimal action can be found by solving
DDPG interleaves learning an approximator to with learning an approximator to , and it does so in a way which is specifically adapted for environments with continuous action spaces. But what does it mean that DDPG is adapted specifically for environments with continuous action spaces? It relates to how we compute the max over actions in .
When there are a finite number of discrete actions, the max poses no problem, because we can just compute the Qvalues for each action separately and directly compare them. (This also immediately gives us the action which maximizes the Qvalue.) But when the action space is continuous, we can’t exhaustively evaluate the space, and solving the optimization problem is highly nontrivial. Using a normal optimization algorithm would make calculating a painfully expensive subroutine. And since it would need to be run every time the agent wants to take an action in the environment, this is unacceptable.
Because the action space is continuous, the function is presumed to be differentiable with respect to the action argument. This allows us to set up an efficient, gradientbased learning rule for a policy which exploits that fact. Then, instead of running an expensive optimization subroutine each time we wish to compute , we can approximate it with . See the Key Equations section details.
Quick FactsÂ¶
 DDPG is an offpolicy algorithm.
 DDPG can only be used for environments with continuous action spaces.
 DDPG can be thought of as being deep Qlearning for continuous action spaces.
 The Spinning Up implementation of DDPG does not support parallelization.
Key EquationsÂ¶
Here, we’ll explain the math behind the two parts of DDPG: learning a Q function, and learning a policy.
The QLearning Side of DDPGÂ¶
First, let’s recap the Bellman equation describing the optimal actionvalue function, . It’s given by
where is shorthand for saying that the next state, , is sampled by the environment from a distribution .
This Bellman equation is the starting point for learning an approximator to . Suppose the approximator is a neural network , with parameters , and that we have collected a set of transitions (where indicates whether state is terminal). We can set up a meansquared Bellman error (MSBE) function, which tells us roughly how closely comes to satisfying the Bellman equation:
Here, in evaluating , we’ve used a Python convention of evaluating True
to 1 and False
to zero. Thus, when d==True
—which is to say, when is a terminal state—the Qfunction should show that the agent gets no additional rewards after the current state. (This choice of notation corresponds to what we later implement in code.)
Qlearning algorithms for function approximators, such as DQN (and all its variants) and DDPG, are largely based on minimizing this MSBE loss function. There are two main tricks employed by all of them which are worth describing, and then a specific detail for DDPG.
Trick One: Replay Buffers. All standard algorithms for training a deep neural network to approximate make use of an experience replay buffer. This is the set of previous experiences. In order for the algorithm to have stable behavior, the replay buffer should be large enough to contain a wide range of experiences, but it may not always be good to keep everything. If you only use the verymost recent data, you will overfit to that and things will break; if you use too much experience, you may slow down your learning. This may take some tuning to get right.
You Should Know
We’ve mentioned that DDPG is an offpolicy algorithm: this is as good a point as any to highlight why and how. Observe that the replay buffer should contain old experiences, even though they might have been obtained using an outdated policy. Why are we able to use these at all? The reason is that the Bellman equation doesn’t care which transition tuples are used, or how the actions were selected, or what happens after a given transition, because the optimal Qfunction should satisfy the Bellman equation for all possible transitions. So any transitions that we’ve ever experienced are fair game when trying to fit a Qfunction approximator via MSBE minimization.
Trick Two: Target Networks. Qlearning algorithms make use of target networks. The term
is called the target, because when we minimize the MSBE loss, we are trying to make the Qfunction be more like this target. Problematically, the target depends on the same parameters we are trying to train: . This makes MSBE minimization unstable. The solution is to use a set of parameters which comes close to , but with a time delay—that is to say, a second network, called the target network, which lags the first. The parameters of the target network are denoted .
In DQNbased algorithms, the target network is just copied over from the main network every somefixednumber of steps. In DDPGstyle algorithms, the target network is updated once per main network update by polyak averaging:
where is a hyperparameter between 0 and 1 (usually close to 1). (This hyperparameter is called polyak
in our code).
DDPG Detail: Calculating the Max Over Actions in the Target. As mentioned earlier: computing the maximum over actions in the target is a challenge in continuous action spaces. DDPG deals with this by using a target policy network to compute an action which approximately maximizes . The target policy network is found the same way as the target Qfunction: by polyak averaging the policy parameters over the course of training.
Putting it all together, Qlearning in DDPG is performed by minimizing the following MSBE loss with stochastic gradient descent:
where is the target policy.
The Policy Learning Side of DDPGÂ¶
Policy learning in DDPG is fairly simple. We want to learn a deterministic policy which gives the action that maximizes . Because the action space is continuous, and we assume the Qfunction is differentiable with respect to action, we can just perform gradient ascent (with respect to policy parameters only) to solve
Note that the Qfunction parameters are treated as constants here.
Exploration vs. ExploitationÂ¶
DDPG trains a deterministic policy in an offpolicy way. Because the policy is deterministic, if the agent were to explore onpolicy, in the beginning it would probably not try a wide enough variety of actions to find useful learning signals. To make DDPG policies explore better, we add noise to their actions at training time. The authors of the original DDPG paper recommended timecorrelated OU noise, but more recent results suggest that uncorrelated, meanzero Gaussian noise works perfectly well. Since the latter is simpler, it is preferred. To facilitate getting higherquality training data, you may reduce the scale of the noise over the course of training. (We do not do this in our implementation, and keep noise scale fixed throughout.)
At test time, to see how well the policy exploits what it has learned, we do not add noise to the actions.
You Should Know
Our DDPG implementation uses a trick to improve exploration at the start of training. For a fixed number of steps at the beginning (set with the start_steps
keyword argument), the agent takes actions which are sampled from a uniform random distribution over valid actions. After that, it returns to normal DDPG exploration.
DocumentationÂ¶
You Should Know
In what follows, we give documentation for the PyTorch and Tensorflow implementations of DDPG in Spinning Up. They have nearly identical function calls and docstrings, except for details relating to model construction. However, we include both full docstrings for completeness.
Documentation: PyTorch VersionÂ¶

spinup.
ddpg_pytorch
(env_fn, actor_critic=<MagicMock spec='str' id='140191853999552'>, ac_kwargs={}, seed=0, steps_per_epoch=4000, epochs=100, replay_size=1000000, gamma=0.99, polyak=0.995, pi_lr=0.001, q_lr=0.001, batch_size=100, start_steps=10000, update_after=1000, update_every=50, act_noise=0.1, num_test_episodes=10, max_ep_len=1000, logger_kwargs={}, save_freq=1)Â¶ Deep Deterministic Policy Gradient (DDPG)
Parameters:  env_fn – A function which creates a copy of the environment. The environment must satisfy the OpenAI Gym API.
 actor_critic –
The constructor method for a PyTorch Module with an
act
method, api
module, and aq
module. Theact
method andpi
module should accept batches of observations as inputs, andq
should accept a batch of observations and a batch of actions as inputs. When called, these should return:Call Output Shape Description act
(batch, act_dim) Numpy array of actions for eachobservation.pi
(batch, act_dim) Tensor containing actions from policygiven observations.q
(batch,) Tensor containing the current estimateof Q* for the provided observationsand actions. (Critical: make sure toflatten this!)  ac_kwargs (dict) – Any kwargs appropriate for the ActorCritic object you provided to DDPG.
 seed (int) – Seed for random number generators.
 steps_per_epoch (int) – Number of steps of interaction (stateaction pairs) for the agent and the environment in each epoch.
 epochs (int) – Number of epochs to run and train agent.
 replay_size (int) – Maximum length of replay buffer.
 gamma (float) – Discount factor. (Always between 0 and 1.)
 polyak (float) –
Interpolation factor in polyak averaging for target networks. Target networks are updated towards main networks according to:
where is polyak. (Always between 0 and 1, usually close to 1.)
 pi_lr (float) – Learning rate for policy.
 q_lr (float) – Learning rate for Qnetworks.
 batch_size (int) – Minibatch size for SGD.
 start_steps (int) – Number of steps for uniformrandom action selection, before running real policy. Helps exploration.
 update_after (int) – Number of env interactions to collect before starting to do gradient descent updates. Ensures replay buffer is full enough for useful updates.
 update_every (int) – Number of env interactions that should elapse between gradient descent updates. Note: Regardless of how long you wait between updates, the ratio of env steps to gradient steps is locked to 1.
 act_noise (float) – Stddev for Gaussian exploration noise added to policy at training time. (At test time, no noise is added.)
 num_test_episodes (int) – Number of episodes to test the deterministic policy at the end of each epoch.
 max_ep_len (int) – Maximum length of trajectory / episode / rollout.
 logger_kwargs (dict) – Keyword args for EpochLogger.
 save_freq (int) – How often (in terms of gap between epochs) to save the current policy and value function.
Saved Model Contents: PyTorch VersionÂ¶
The PyTorch saved model can be loaded with ac = torch.load('path/to/model.pt')
, yielding an actorcritic object (ac
) that has the properties described in the docstring for ddpg_pytorch
.
You can get actions from this model with
actions = ac.act(torch.as_tensor(obs, dtype=torch.float32))
Documentation: Tensorflow VersionÂ¶

spinup.
ddpg_tf1
(env_fn, actor_critic=<function mlp_actor_critic>, ac_kwargs={}, seed=0, steps_per_epoch=4000, epochs=100, replay_size=1000000, gamma=0.99, polyak=0.995, pi_lr=0.001, q_lr=0.001, batch_size=100, start_steps=10000, update_after=1000, update_every=50, act_noise=0.1, num_test_episodes=10, max_ep_len=1000, logger_kwargs={}, save_freq=1)Â¶ Deep Deterministic Policy Gradient (DDPG)
Parameters:  env_fn – A function which creates a copy of the environment. The environment must satisfy the OpenAI Gym API.
 actor_critic –
A function which takes in placeholder symbols for state,
x_ph
, and action,a_ph
, and returns the main outputs from the agent’s Tensorflow computation graph:Symbol Shape Description pi
(batch, act_dim) Deterministically computes actionsfrom policy given states.q
(batch,) Gives the current estimate of Q* forstates inx_ph
and actions ina_ph
.q_pi
(batch,) Gives the composition ofq
andpi
for states inx_ph
:q(x, pi(x)).  ac_kwargs (dict) – Any kwargs appropriate for the actor_critic function you provided to DDPG.
 seed (int) – Seed for random number generators.
 steps_per_epoch (int) – Number of steps of interaction (stateaction pairs) for the agent and the environment in each epoch.
 epochs (int) – Number of epochs to run and train agent.
 replay_size (int) – Maximum length of replay buffer.
 gamma (float) – Discount factor. (Always between 0 and 1.)
 polyak (float) –
Interpolation factor in polyak averaging for target networks. Target networks are updated towards main networks according to:
where is polyak. (Always between 0 and 1, usually close to 1.)
 pi_lr (float) – Learning rate for policy.
 q_lr (float) – Learning rate for Qnetworks.
 batch_size (int) – Minibatch size for SGD.
 start_steps (int) – Number of steps for uniformrandom action selection, before running real policy. Helps exploration.
 update_after (int) – Number of env interactions to collect before starting to do gradient descent updates. Ensures replay buffer is full enough for useful updates.
 update_every (int) – Number of env interactions that should elapse between gradient descent updates. Note: Regardless of how long you wait between updates, the ratio of env steps to gradient steps is locked to 1.
 act_noise (float) – Stddev for Gaussian exploration noise added to policy at training time. (At test time, no noise is added.)
 num_test_episodes (int) – Number of episodes to test the deterministic policy at the end of each epoch.
 max_ep_len (int) – Maximum length of trajectory / episode / rollout.
 logger_kwargs (dict) – Keyword args for EpochLogger.
 save_freq (int) – How often (in terms of gap between epochs) to save the current policy and value function.
Saved Model Contents: Tensorflow VersionÂ¶
The computation graph saved by the logger includes:
Key  Value 

x 
Tensorflow placeholder for state input. 
a 
Tensorflow placeholder for action input. 
pi 
Deterministically computes an action from the agent, conditioned
on states in
x . 
q 
Gives actionvalue estimate for states in x and actions in a . 
This saved model can be accessed either by
 running the trained policy with the test_policy.py tool,
 or loading the whole saved graph into a program with restore_tf_graph.
ReferencesÂ¶
Relevant PapersÂ¶
 Deterministic Policy Gradient Algorithms, Silver et al. 2014
 Continuous Control With Deep Reinforcement Learning, Lillicrap et al. 2016
Why These Papers?Â¶
Silver 2014 is included because it establishes the theory underlying deterministic policy gradients (DPG). Lillicrap 2016 is included because it adapts the theoreticallygrounded DPG algorithm to the deep RL setting, giving DDPG.
Twin Delayed DDPGÂ¶
Table of Contents
BackgroundÂ¶
(Previously: Background for DDPG)
While DDPG can achieve great performance sometimes, it is frequently brittle with respect to hyperparameters and other kinds of tuning. A common failure mode for DDPG is that the learned Qfunction begins to dramatically overestimate Qvalues, which then leads to the policy breaking, because it exploits the errors in the Qfunction. Twin Delayed DDPG (TD3) is an algorithm that addresses this issue by introducing three critical tricks:
Trick One: Clipped DoubleQ Learning. TD3 learns two Qfunctions instead of one (hence “twin”), and uses the smaller of the two Qvalues to form the targets in the Bellman error loss functions.
Trick Two: “Delayed” Policy Updates. TD3 updates the policy (and target networks) less frequently than the Qfunction. The paper recommends one policy update for every two Qfunction updates.
Trick Three: Target Policy Smoothing. TD3 adds noise to the target action, to make it harder for the policy to exploit Qfunction errors by smoothing out Q along changes in action.
Together, these three tricks result in substantially improved performance over baseline DDPG.
Quick FactsÂ¶
 TD3 is an offpolicy algorithm.
 TD3 can only be used for environments with continuous action spaces.
 The Spinning Up implementation of TD3 does not support parallelization.
Key EquationsÂ¶
TD3 concurrently learns two Qfunctions, and , by mean square Bellman error minimization, in almost the same way that DDPG learns its single Qfunction. To show exactly how TD3 does this and how it differs from normal DDPG, we’ll work from the innermost part of the loss function outwards.
First: target policy smoothing. Actions used to form the Qlearning target are based on the target policy, , but with clipped noise added on each dimension of the action. After adding the clipped noise, the target action is then clipped to lie in the valid action range (all valid actions, , satisfy ). The target actions are thus:
Target policy smoothing essentially serves as a regularizer for the algorithm. It addresses a particular failure mode that can happen in DDPG: if the Qfunction approximator develops an incorrect sharp peak for some actions, the policy will quickly exploit that peak and then have brittle or incorrect behavior. This can be averted by smoothing out the Qfunction over similar actions, which target policy smoothing is designed to do.
Next: clipped doubleQ learning. Both Qfunctions use a single target, calculated using whichever of the two Qfunctions gives a smaller target value:
and then both are learned by regressing to this target:
Using the smaller Qvalue for the target, and regressing towards that, helps fend off overestimation in the Qfunction.
Lastly: the policy is learned just by maximizing :
which is pretty much unchanged from DDPG. However, in TD3, the policy is updated less frequently than the Qfunctions are. This helps damp the volatility that normally arises in DDPG because of how a policy update changes the target.
Exploration vs. ExploitationÂ¶
TD3 trains a deterministic policy in an offpolicy way. Because the policy is deterministic, if the agent were to explore onpolicy, in the beginning it would probably not try a wide enough variety of actions to find useful learning signals. To make TD3 policies explore better, we add noise to their actions at training time, typically uncorrelated meanzero Gaussian noise. To facilitate getting higherquality training data, you may reduce the scale of the noise over the course of training. (We do not do this in our implementation, and keep noise scale fixed throughout.)
At test time, to see how well the policy exploits what it has learned, we do not add noise to the actions.
You Should Know
Our TD3 implementation uses a trick to improve exploration at the start of training. For a fixed number of steps at the beginning (set with the start_steps
keyword argument), the agent takes actions which are sampled from a uniform random distribution over valid actions. After that, it returns to normal TD3 exploration.
DocumentationÂ¶
You Should Know
In what follows, we give documentation for the PyTorch and Tensorflow implementations of TD3 in Spinning Up. They have nearly identical function calls and docstrings, except for details relating to model construction. However, we include both full docstrings for completeness.
Documentation: PyTorch VersionÂ¶

spinup.
td3_pytorch
(env_fn, actor_critic=<MagicMock spec='str' id='140191853475208'>, ac_kwargs={}, seed=0, steps_per_epoch=4000, epochs=100, replay_size=1000000, gamma=0.99, polyak=0.995, pi_lr=0.001, q_lr=0.001, batch_size=100, start_steps=10000, update_after=1000, update_every=50, act_noise=0.1, target_noise=0.2, noise_clip=0.5, policy_delay=2, num_test_episodes=10, max_ep_len=1000, logger_kwargs={}, save_freq=1)Â¶ Twin Delayed Deep Deterministic Policy Gradient (TD3)
Parameters:  env_fn – A function which creates a copy of the environment. The environment must satisfy the OpenAI Gym API.
 actor_critic –
The constructor method for a PyTorch Module with an
act
method, api
module, aq1
module, and aq2
module. Theact
method andpi
module should accept batches of observations as inputs, andq1
andq2
should accept a batch of observations and a batch of actions as inputs. When called, these should return:Call Output Shape Description act
(batch, act_dim) Numpy array of actions for eachobservation.pi
(batch, act_dim) Tensor containing actions from policygiven observations.q1
(batch,) Tensor containing one current estimateof Q* for the provided observationsand actions. (Critical: make sure toflatten this!)q2
(batch,) Tensor containing the other currentestimate of Q* for the provided observationsand actions. (Critical: make sure toflatten this!)  ac_kwargs (dict) – Any kwargs appropriate for the ActorCritic object you provided to TD3.
 seed (int) – Seed for random number generators.
 steps_per_epoch (int) – Number of steps of interaction (stateaction pairs) for the agent and the environment in each epoch.
 epochs (int) – Number of epochs to run and train agent.
 replay_size (int) – Maximum length of replay buffer.
 gamma (float) – Discount factor. (Always between 0 and 1.)
 polyak (float) –
Interpolation factor in polyak averaging for target networks. Target networks are updated towards main networks according to:
where is polyak. (Always between 0 and 1, usually close to 1.)
 pi_lr (float) – Learning rate for policy.
 q_lr (float) – Learning rate for Qnetworks.
 batch_size (int) – Minibatch size for SGD.
 start_steps (int) – Number of steps for uniformrandom action selection, before running real policy. Helps exploration.
 update_after (int) – Number of env interactions to collect before starting to do gradient descent updates. Ensures replay buffer is full enough for useful updates.
 update_every (int) – Number of env interactions that should elapse between gradient descent updates. Note: Regardless of how long you wait between updates, the ratio of env steps to gradient steps is locked to 1.
 act_noise (float) – Stddev for Gaussian exploration noise added to policy at training time. (At test time, no noise is added.)
 target_noise (float) – Stddev for smoothing noise added to target policy.
 noise_clip (float) – Limit for absolute value of target policy smoothing noise.
 policy_delay (int) – Policy will only be updated once every policy_delay times for each update of the Qnetworks.
 num_test_episodes (int) – Number of episodes to test the deterministic policy at the end of each epoch.
 max_ep_len (int) – Maximum length of trajectory / episode / rollout.
 logger_kwargs (dict) – Keyword args for EpochLogger.
 save_freq (int) – How often (in terms of gap between epochs) to save the current policy and value function.
Saved Model Contents: PyTorch VersionÂ¶
The PyTorch saved model can be loaded with ac = torch.load('path/to/model.pt')
, yielding an actorcritic object (ac
) that has the properties described in the docstring for td3_pytorch
.
You can get actions from this model with
actions = ac.act(torch.as_tensor(obs, dtype=torch.float32))
Documentation: Tensorflow VersionÂ¶

spinup.
td3_tf1
(env_fn, actor_critic=<function mlp_actor_critic>, ac_kwargs={}, seed=0, steps_per_epoch=4000, epochs=100, replay_size=1000000, gamma=0.99, polyak=0.995, pi_lr=0.001, q_lr=0.001, batch_size=100, start_steps=10000, update_after=1000, update_every=50, act_noise=0.1, target_noise=0.2, noise_clip=0.5, policy_delay=2, num_test_episodes=10, max_ep_len=1000, logger_kwargs={}, save_freq=1)Â¶ Twin Delayed Deep Deterministic Policy Gradient (TD3)
Parameters:  env_fn – A function which creates a copy of the environment. The environment must satisfy the OpenAI Gym API.
 actor_critic –
A function which takes in placeholder symbols for state,
x_ph
, and action,a_ph
, and returns the main outputs from the agent’s Tensorflow computation graph:Symbol Shape Description pi
(batch, act_dim) Deterministically computes actionsfrom policy given states.q1
(batch,) Gives one estimate of Q* forstates inx_ph
and actions ina_ph
.q2
(batch,) Gives another estimate of Q* forstates inx_ph
and actions ina_ph
.q1_pi
(batch,) Gives the composition ofq1
andpi
for states inx_ph
:q1(x, pi(x)).  ac_kwargs (dict) – Any kwargs appropriate for the actor_critic function you provided to TD3.
 seed (int) – Seed for random number generators.
 steps_per_epoch (int) – Number of steps of interaction (stateaction pairs) for the agent and the environment in each epoch.
 epochs (int) – Number of epochs to run and train agent.
 replay_size (int) – Maximum length of replay buffer.
 gamma (float) – Discount factor. (Always between 0 and 1.)
 polyak (float) –
Interpolation factor in polyak averaging for target networks. Target networks are updated towards main networks according to:
where is polyak. (Always between 0 and 1, usually close to 1.)
 pi_lr (float) – Learning rate for policy.
 q_lr (float) – Learning rate for Qnetworks.
 batch_size (int) – Minibatch size for SGD.
 start_steps (int) – Number of steps for uniformrandom action selection, before running real policy. Helps exploration.
 update_after (int) – Number of env interactions to collect before starting to do gradient descent updates. Ensures replay buffer is full enough for useful updates.
 update_every (int) – Number of env interactions that should elapse between gradient descent updates. Note: Regardless of how long you wait between updates, the ratio of env steps to gradient steps is locked to 1.
 act_noise (float) – Stddev for Gaussian exploration noise added to policy at training time. (At test time, no noise is added.)
 target_noise (float) – Stddev for smoothing noise added to target policy.
 noise_clip (float) – Limit for absolute value of target policy smoothing noise.
 policy_delay (int) – Policy will only be updated once every policy_delay times for each update of the Qnetworks.
 num_test_episodes (int) – Number of episodes to test the deterministic policy at the end of each epoch.
 max_ep_len (int) – Maximum length of trajectory / episode / rollout.
 logger_kwargs (dict) – Keyword args for EpochLogger.
 save_freq (int) – How often (in terms of gap between epochs) to save the current policy and value function.
Saved Model Contents: Tensorflow VersionÂ¶
The computation graph saved by the logger includes:
Key  Value 

x 
Tensorflow placeholder for state input. 
a 
Tensorflow placeholder for action input. 
pi 
Deterministically computes an action from the agent, conditioned
on states in
x . 
q1 
Gives one actionvalue estimate for states in x and actions in a . 
q2 
Gives the other actionvalue estimate for states in x and actions in a . 
This saved model can be accessed either by
 running the trained policy with the test_policy.py tool,
 or loading the whole saved graph into a program with restore_tf_graph.
Soft ActorCriticÂ¶
Table of Contents
BackgroundÂ¶
(Previously: Background for TD3)
Soft Actor Critic (SAC) is an algorithm that optimizes a stochastic policy in an offpolicy way, forming a bridge between stochastic policy optimization and DDPGstyle approaches. It isn’t a direct successor to TD3 (having been published roughly concurrently), but it incorporates the clipped doubleQ trick, and due to the inherent stochasticity of the policy in SAC, it also winds up benefiting from something like target policy smoothing.
A central feature of SAC is entropy regularization. The policy is trained to maximize a tradeoff between expected return and entropy, a measure of randomness in the policy. This has a close connection to the explorationexploitation tradeoff: increasing entropy results in more exploration, which can accelerate learning later on. It can also prevent the policy from prematurely converging to a bad local optimum.
Quick FactsÂ¶
 SAC is an offpolicy algorithm.
 The version of SAC implemented here can only be used for environments with continuous action spaces.
 An alternate version of SAC, which slightly changes the policy update rule, can be implemented to handle discrete action spaces.
 The Spinning Up implementation of SAC does not support parallelization.
Key EquationsÂ¶
To explain Soft Actor Critic, we first have to introduce the entropyregularized reinforcement learning setting. In entropyregularized RL, there are slightlydifferent equations for value functions.
EntropyRegularized Reinforcement LearningÂ¶
Entropy is a quantity which, roughly speaking, says how random a random variable is. If a coin is weighted so that it almost always comes up heads, it has low entropy; if it’s evenly weighted and has a half chance of either outcome, it has high entropy.
Let be a random variable with probability mass or density function . The entropy of is computed from its distribution according to
In entropyregularized reinforcement learning, the agent gets a bonus reward at each time step proportional to the entropy of the policy at that timestep. This changes the RL problem to:
where is the tradeoff coefficient. (Note: we’re assuming an infinitehorizon discounted setting here, and we’ll do the same for the rest of this page.) We can now define the slightlydifferent value functions in this setting. is changed to include the entropy bonuses from every timestep:
is changed to include the entropy bonuses from every timestep except the first:
With these definitions, and are connected by:
and the Bellman equation for is
You Should Know
The way we’ve set up the value functions in the entropyregularized setting is a little bit arbitrary, and actually we could have done it differently (eg make include the entropy bonus at the first timestep). The choice of definition may vary slightly across papers on the subject.
Soft ActorCriticÂ¶
SAC concurrently learns a policy and two Qfunctions . There are two variants of SAC that are currently standard: one that uses a fixed entropy regularization coefficient , and another that enforces an entropy constraint by varying over the course of training. For simplicity, Spinning Up makes use of the version with a fixed entropy regularization coefficient, but the entropyconstrained variant is generally preferred by practitioners.
You Should Know
The SAC algorithm has changed a little bit over time. An older version of SAC also learns a value function in addition to the Qfunctions; this page will focus on the modern version that omits the extra value function.
Learning Q. The Qfunctions are learned in a similar way to TD3, but with a few key differences.
First, what’s similar?
 Like in TD3, both Qfunctions are learned with MSBE minimization, by regressing to a single shared target.
 Like in TD3, the shared target is computed using target Qnetworks, and the target Qnetworks are obtained by polyak averaging the Qnetwork parameters over the course of training.
 Like in TD3, the shared target makes use of the clipped doubleQ trick.
What’s different?
 Unlike in TD3, the target also includes a term that comes from SAC’s use of entropy regularization.
 Unlike in TD3, the nextstate actions used in the target come from the current policy instead of a target policy.
 Unlike in TD3, there is no explicit target policy smoothing. TD3 trains a deterministic policy, and so it accomplishes smoothing by adding random noise to the nextstate actions. SAC trains a stochastic policy, and so the noise from that stochasticity is sufficient to get a similar effect.
Before we give the final form of the Qloss, letâ€™s take a moment to discuss how the contribution from entropy regularization comes in. We’ll start by taking our recursive Bellman equation for the entropyregularized from earlier, and rewriting it a little bit by using the definition of entropy:
The RHS is an expectation over next states (which come from the replay buffer) and next actions (which come from the current policy, and not the replay buffer). Since it’s an expectation, we can approximate it with samples:
You Should Know
We switch next action notation to , instead of , to highlight that the next actions have to be sampled fresh from the policy (whereas by contrast, and should come from the replay buffer).
SAC sets up the MSBE loss for each Qfunction using this kind of sample approximation for the target. The only thing still undetermined here is which Qfunction gets used to compute the sample backup: like TD3, SAC uses the clipped doubleQ trick, and takes the minimum Qvalue between the two Q approximators.
Putting it all together, the loss functions for the Qnetworks in SAC are:
where the target is given by
Learning the Policy. The policy should, in each state, act to maximize the expected future return plus expected future entropy. That is, it should maximize , which we expand out into
The way we optimize the policy makes use of the reparameterization trick, in which a sample from is drawn by computing a deterministic function of state, policy parameters, and independent noise. To illustrate: following the authors of the SAC paper, we use a squashed Gaussian policy, which means that samples are obtained according to
You Should Know
This policy has two key differences from the policies we use in the other policy optimization algorithms:
1. The squashing function. The in the SAC policy ensures that actions are bounded to a finite range. This is absent in the VPG, TRPO, and PPO policies. It also changes the distribution: before the the SAC policy is a factored Gaussian like the other algorithms’ policies, but after the it is not. (You can still compute the logprobabilities of actions in closed form, though: see the paper appendix for details.)
2. The way standard deviations are parameterized. In VPG, TRPO, and PPO, we represent the log std devs with stateindependent parameter vectors. In SAC, we represent the log std devs as outputs from the neural network, meaning that they depend on state in a complex way. SAC with stateindependent log std devs, in our experience, did not work. (Can you think of why? Or better yet: run an experiment to verify?)
The reparameterization trick allows us to rewrite the expectation over actions (which contains a pain point: the distribution depends on the policy parameters) into an expectation over noise (which removes the pain point: the distribution now has no dependence on parameters):
To get the policy loss, the final step is that we need to substitute with one of our function approximators. Unlike in TD3, which uses (just the first Q approximator), SAC uses (the minimum of the two Q approximators). The policy is thus optimized according to
which is almost the same as the DDPG and TD3 policy optimization, except for the mindoubleQ trick, the stochasticity, and the entropy term.
Exploration vs. ExploitationÂ¶
SAC trains a stochastic policy with entropy regularization, and explores in an onpolicy way. The entropy regularization coefficient explicitly controls the exploreexploit tradeoff, with higher corresponding to more exploration, and lower corresponding to more exploitation. The right coefficient (the one which leads to the stablest / highestreward learning) may vary from environment to environment, and could require careful tuning.
At test time, to see how well the policy exploits what it has learned, we remove stochasticity and use the mean action instead of a sample from the distribution. This tends to improve performance over the original stochastic policy.
You Should Know
Our SAC implementation uses a trick to improve exploration at the start of training. For a fixed number of steps at the beginning (set with the start_steps
keyword argument), the agent takes actions which are sampled from a uniform random distribution over valid actions. After that, it returns to normal SAC exploration.
DocumentationÂ¶
You Should Know
In what follows, we give documentation for the PyTorch and Tensorflow implementations of SAC in Spinning Up. They have nearly identical function calls and docstrings, except for details relating to model construction. However, we include both full docstrings for completeness.
Documentation: PyTorch VersionÂ¶

spinup.
sac_pytorch
(env_fn, actor_critic=<MagicMock spec='str' id='140191856645456'>, ac_kwargs={}, seed=0, steps_per_epoch=4000, epochs=100, replay_size=1000000, gamma=0.99, polyak=0.995, lr=0.001, alpha=0.2, batch_size=100, start_steps=10000, update_after=1000, update_every=50, num_test_episodes=10, max_ep_len=1000, logger_kwargs={}, save_freq=1)Â¶ Soft ActorCritic (SAC)
Parameters:  env_fn – A function which creates a copy of the environment. The environment must satisfy the OpenAI Gym API.
 actor_critic –
The constructor method for a PyTorch Module with an
act
method, api
module, aq1
module, and aq2
module. Theact
method andpi
module should accept batches of observations as inputs, andq1
andq2
should accept a batch of observations and a batch of actions as inputs. When called,act
,q1
, andq2
should return:Call Output Shape Description act
(batch, act_dim) Numpy array of actions for eachobservation.q1
(batch,) Tensor containing one current estimateof Q* for the provided observationsand actions. (Critical: make sure toflatten this!)q2
(batch,) Tensor containing the other currentestimate of Q* for the provided observationsand actions. (Critical: make sure toflatten this!)Calling
pi
should return:Symbol Shape Description a
(batch, act_dim) Tensor containing actions from policygiven observations.logp_pi
(batch,) Tensor containing log probabilities ofactions ina
. Importantly: gradientsshould be able to flow back intoa
.  ac_kwargs (dict) – Any kwargs appropriate for the ActorCritic object you provided to SAC.
 seed (int) – Seed for random number generators.
 steps_per_epoch (int) – Number of steps of interaction (stateaction pairs) for the agent and the environment in each epoch.
 epochs (int) – Number of epochs to run and train agent.
 replay_size (int) – Maximum length of replay buffer.
 gamma (float) – Discount factor. (Always between 0 and 1.)
 polyak (float) –
Interpolation factor in polyak averaging for target networks. Target networks are updated towards main networks according to:
where is polyak. (Always between 0 and 1, usually close to 1.)
 lr (float) – Learning rate (used for both policy and value learning).
 alpha (float) – Entropy regularization coefficient. (Equivalent to inverse of reward scale in the original SAC paper.)
 batch_size (int) – Minibatch size for SGD.
 start_steps (int) – Number of steps for uniformrandom action selection, before running real policy. Helps exploration.
 update_after (int) – Number of env interactions to collect before starting to do gradient descent updates. Ensures replay buffer is full enough for useful updates.
 update_every (int) – Number of env interactions that should elapse between gradient descent updates. Note: Regardless of how long you wait between updates, the ratio of env steps to gradient steps is locked to 1.
 num_test_episodes (int) – Number of episodes to test the deterministic policy at the end of each epoch.
 max_ep_len (int) – Maximum length of trajectory / episode / rollout.
 logger_kwargs (dict) – Keyword args for EpochLogger.
 save_freq (int) – How often (in terms of gap between epochs) to save the current policy and value function.
Saved Model Contents: PyTorch VersionÂ¶
The PyTorch saved model can be loaded with ac = torch.load('path/to/model.pt')
, yielding an actorcritic object (ac
) that has the properties described in the docstring for sac_pytorch
.
You can get actions from this model with
actions = ac.act(torch.as_tensor(obs, dtype=torch.float32))
Documentation: Tensorflow VersionÂ¶

spinup.
sac_tf1
(env_fn, actor_critic=<function mlp_actor_critic>, ac_kwargs={}, seed=0, steps_per_epoch=4000, epochs=100, replay_size=1000000, gamma=0.99, polyak=0.995, lr=0.001, alpha=0.2, batch_size=100, start_steps=10000, update_after=1000, update_every=50, num_test_episodes=10, max_ep_len=1000, logger_kwargs={}, save_freq=1)Â¶ Soft ActorCritic (SAC)
Parameters:  env_fn – A function which creates a copy of the environment. The environment must satisfy the OpenAI Gym API.
 actor_critic –
A function which takes in placeholder symbols for state,
x_ph
, and action,a_ph
, and returns the main outputs from the agent’s Tensorflow computation graph:Symbol Shape Description mu
(batch, act_dim) Computes mean actions from policygiven states.pi
(batch, act_dim) Samples actions from policy givenstates.logp_pi
(batch,) Gives log probability, according tothe policy, of the action sampled bypi
. Critical: must be differentiablewith respect to policy parameters allthe way through action sampling.q1
(batch,) Gives one estimate of Q* forstates inx_ph
and actions ina_ph
.q2
(batch,) Gives another estimate of Q* forstates inx_ph
and actions ina_ph
.  ac_kwargs (dict) – Any kwargs appropriate for the actor_critic function you provided to SAC.
 seed (int) – Seed for random number generators.
 steps_per_epoch (int) – Number of steps of interaction (stateaction pairs) for the agent and the environment in each epoch.
 epochs (int) – Number of epochs to run and train agent.
 replay_size (int) – Maximum length of replay buffer.
 gamma (float) – Discount factor. (Always between 0 and 1.)
 polyak (float) –
Interpolation factor in polyak averaging for target networks. Target networks are updated towards main networks according to:
where is polyak. (Always between 0 and 1, usually close to 1.)
 lr (float) – Learning rate (used for both policy and value learning).
 alpha (float) – Entropy regularization coefficient. (Equivalent to inverse of reward scale in the original SAC paper.)
 batch_size (int) – Minibatch size for SGD.
 start_steps (int) – Number of steps for uniformrandom action selection, before running real policy. Helps exploration.
 update_after (int) – Number of env interactions to collect before starting to do gradient descent updates. Ensures replay buffer is full enough for useful updates.
 update_every (int) – Number of env interactions that should elapse between gradient descent updates. Note: Regardless of how long you wait between updates, the ratio of env steps to gradient steps is locked to 1.
 num_test_episodes (int) – Number of episodes to test the deterministic policy at the end of each epoch.
 max_ep_len (int) – Maximum length of trajectory / episode / rollout.
 logger_kwargs (dict) – Keyword args for EpochLogger.
 save_freq (int) – How often (in terms of gap between epochs) to save the current policy and value function.
Saved Model Contents: Tensorflow VersionÂ¶
The computation graph saved by the logger includes:
Key  Value 

x 
Tensorflow placeholder for state input. 
a 
Tensorflow placeholder for action input. 
mu 
Deterministically computes mean action from the agent, given states in x . 
pi 
Samples an action from the agent, conditioned on states in x . 
q1 
Gives one actionvalue estimate for states in x and actions in a . 
q2 
Gives the other actionvalue estimate for states in x and actions in a . 
v 
Gives the value estimate for states in x . 
This saved model can be accessed either by
 running the trained policy with the test_policy.py tool,
 or loading the whole saved graph into a program with restore_tf_graph.
Note: for SAC, the correct evaluation policy is given by mu
and not by pi
. The policy pi
may be thought of as the exploration policy, while mu
is the exploitation policy.
ReferencesÂ¶
Relevant PapersÂ¶
 Soft ActorCritic: OffPolicy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor, Haarnoja et al, 2018
 Soft ActorCritic Algorithms and Applications, Haarnoja et al, 2018
 Learning to Walk via Deep Reinforcement Learning, Haarnoja et al, 2018
Other Public ImplementationsÂ¶
 SAC release repo (original “official” codebase)
 Softlearning repo (current “official” codebase)
 Yarats and Kostrikov repo
LoggerÂ¶
Table of Contents
Using a LoggerÂ¶
Spinning Up ships with basic logging tools, implemented in the classes Logger and EpochLogger. The Logger class contains most of the basic functionality for saving diagnostics, hyperparameter configurations, the state of a training run, and the trained model. The EpochLogger class adds a thin layer on top of that to make it easy to track the average, standard deviation, min, and max value of a diagnostic over each epoch and across MPI workers.
You Should Know
All Spinning Up algorithm implementations use an EpochLogger.
ExamplesÂ¶
First, let’s look at a simple example of how an EpochLogger keeps track of a diagnostic value:
>>> from spinup.utils.logx import EpochLogger
>>> epoch_logger = EpochLogger()
>>> for i in range(10):
epoch_logger.store(Test=i)
>>> epoch_logger.log_tabular('Test', with_min_and_max=True)
>>> epoch_logger.dump_tabular()

 AverageTest  4.5 
 StdTest  2.87 
 MaxTest  9 
 MinTest  0 

The store
method is used to save all values of Test
to the epoch_logger
‘s internal state. Then, when log_tabular
is called, it computes the average, standard deviation, min, and max of Test
over all of the values in the internal state. The internal state is wiped clean after the call to log_tabular
(to prevent leakage into the statistics at the next epoch). Finally, dump_tabular
is called to write the diagnostics to file and to stdout.
Next, let’s look at a full training procedure with the logger embedded, to highlight configuration and model saving as well as diagnostic logging:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69  import numpy as np
import tensorflow as tf
import time
from spinup.utils.logx import EpochLogger
def mlp(x, hidden_sizes=(32,), activation=tf.tanh, output_activation=None):
for h in hidden_sizes[:1]:
x = tf.layers.dense(x, units=h, activation=activation)
return tf.layers.dense(x, units=hidden_sizes[1], activation=output_activation)
# Simple script for training an MLP on MNIST.
def train_mnist(steps_per_epoch=100, epochs=5,
lr=1e3, layers=2, hidden_size=64,
logger_kwargs=dict(), save_freq=1):
logger = EpochLogger(**logger_kwargs)
logger.save_config(locals())
# Load and preprocess MNIST data
(x_train, y_train), _ = tf.keras.datasets.mnist.load_data()
x_train = x_train.reshape(1, 28*28) / 255.0
# Define inputs & main outputs from computation graph
x_ph = tf.placeholder(tf.float32, shape=(None, 28*28))
y_ph = tf.placeholder(tf.int32, shape=(None,))
logits = mlp(x_ph, hidden_sizes=[hidden_size]*layers + [10], activation=tf.nn.relu)
predict = tf.argmax(logits, axis=1, output_type=tf.int32)
# Define loss function, accuracy, and training op
y = tf.one_hot(y_ph, 10)
loss = tf.losses.softmax_cross_entropy(y, logits)
acc = tf.reduce_mean(tf.cast(tf.equal(y_ph, predict), tf.float32))
train_op = tf.train.AdamOptimizer().minimize(loss)
# Prepare session
sess = tf.Session()
sess.run(tf.global_variables_initializer())
# Setup model saving
logger.setup_tf_saver(sess, inputs={'x': x_ph},
outputs={'logits': logits, 'predict': predict})
start_time = time.time()
# Run main training loop
for epoch in range(epochs):
for t in range(steps_per_epoch):
idxs = np.random.randint(0, len(x_train), 32)
feed_dict = {x_ph: x_train[idxs],
y_ph: y_train[idxs]}
outs = sess.run([loss, acc, train_op], feed_dict=feed_dict)
logger.store(Loss=outs[0], Acc=outs[1])
# Save model
if (epoch % save_freq == 0) or (epoch == epochs1):
logger.save_state(state_dict=dict(), itr=None)
# Log info about epoch
logger.log_tabular('Epoch', epoch)
logger.log_tabular('Acc', with_min_and_max=True)
logger.log_tabular('Loss', average_only=True)
logger.log_tabular('TotalGradientSteps', (epoch+1)*steps_per_epoch)
logger.log_tabular('Time', time.time()start_time)
logger.dump_tabular()
if __name__ == '__main__':
train_mnist()

In this example, observe that
 On line 19, logger.save_config is used to save the hyperparameter configuration to a JSON file.
 On lines 42 and 43, logger.setup_tf_saver is used to prepare the logger to save the key elements of the computation graph.
 On line 54, diagnostics are saved to the logger’s internal state via logger.store.
 On line 58, the computation graph is saved once per epoch via logger.save_state.
 On lines 6166, logger.log_tabular and logger.dump_tabular are used to write the epoch diagnostics to file. Note that the keys passed into logger.log_tabular are the same as the keys passed into logger.store.
Logging and PyTorchÂ¶
The preceding example was given in Tensorflow. For PyTorch, everything is the same except for L4243: instead of logger.setup_tf_saver
, you would use logger.setup_pytorch_saver
, and you would pass it a PyTorch module (the network you are training) as an argument.
The behavior of logger.save_state
is the same as in the Tensorflow case: each time it is called, it’ll save the latest version of the PyTorch module.
Logging and MPIÂ¶
You Should Know
Several algorithms in RL are easily parallelized by using MPI to average gradients and/or other key quantities. The Spinning Up loggers are designed to be wellbehaved when using MPI: things will only get written to stdout and to file from the process with rank 0. But information from other processes isn’t lost if you use the EpochLogger: everything which is passed into EpochLogger via store
, regardless of which process it’s stored in, gets used to compute average/std/min/max values for a diagnostic.
Logger ClassesÂ¶

class
spinup.utils.logx.
Logger
(output_dir=None, output_fname='progress.txt', exp_name=None)[source]Â¶ A generalpurpose logger.
Makes it easy to save diagnostics, hyperparameter configurations, the state of a training run, and the trained model.

__init__
(output_dir=None, output_fname='progress.txt', exp_name=None)[source]Â¶ Initialize a Logger.
Parameters:  output_dir (string) – A directory for saving results to. If
None
, defaults to a temp directory of the form/tmp/experiments/somerandomnumber
.  output_fname (string) – Name for the tabseparatedvalue file
containing metrics logged throughout a training run.
Defaults to
progress.txt
.  exp_name (string) – Experiment name. If you run multiple training
runs and give them all the same
exp_name
, the plotter will know to group them. (Use case: if you run the same hyperparameter configuration with multiple random seeds, you should give them all the sameexp_name
.)
 output_dir (string) – A directory for saving results to. If

dump_tabular
()[source]Â¶ Write all of the diagnostics from the current iteration.
Writes both to stdout, and to the output file.

log_tabular
(key, val)[source]Â¶ Log a value of some diagnostic.
Call this only once for each diagnostic quantity, each iteration. After using
log_tabular
to store values for each diagnostic, make sure to calldump_tabular
to write them out to file and stdout (otherwise they will not get saved anywhere).

save_config
(config)[source]Â¶ Log an experiment configuration.
Call this once at the top of your experiment, passing in all important config vars as a dict. This will serialize the config to JSON, while handling anything which can’t be serialized in a graceful way (writing as informative a string as possible).
Example use:
logger = EpochLogger(**logger_kwargs) logger.save_config(locals())

save_state
(state_dict, itr=None)[source]Â¶ Saves the state of an experiment.
To be clear: this is about saving state, not logging diagnostics. All diagnostic logging is separate from this function. This function will save whatever is in
state_dict
—usually just a copy of the environment—and the most recent parameters for the model you previously set up saving for withsetup_tf_saver
.Call with any frequency you prefer. If you only want to maintain a single state and overwrite it at each call with the most recent version, leave
itr=None
. If you want to keep all of the states you save, provide unique (increasing) values for ‘itr’.Parameters:  state_dict (dict) – Dictionary containing essential elements to describe the current state of training.
 itr – An int, or None. Current iteration of training.

setup_pytorch_saver
(what_to_save)[source]Â¶ Set up easy model saving for a single PyTorch model.
Because PyTorch saving and loading is especially painless, this is very minimal; we just need references to whatever we would like to pickle. This is integrated into the logger because the logger knows where the user would like to save information about this training run.
Parameters: what_to_save – Any PyTorch model or serializable object containing PyTorch models.

setup_tf_saver
(sess, inputs, outputs)[source]Â¶ Set up easy model saving for tensorflow.
Call once, after defining your computation graph but before training.
Parameters:  sess – The Tensorflow session in which you train your computation graph.
 inputs (dict) – A dictionary that maps from keys of your choice to the tensorflow placeholders that serve as inputs to the computation graph. Make sure that all of the placeholders needed for your outputs are included!
 outputs (dict) – A dictionary that maps from keys of your choice to the outputs from your computation graph.


class
spinup.utils.logx.
EpochLogger
(*args, **kwargs)[source]Â¶ Bases:
spinup.utils.logx.Logger
A variant of Logger tailored for tracking average values over epochs.
Typical use case: there is some quantity which is calculated many times throughout an epoch, and at the end of the epoch, you would like to report the average / std / min / max value of that quantity.
With an EpochLogger, each time the quantity is calculated, you would use
epoch_logger.store(NameOfQuantity=quantity_value)
to load it into the EpochLogger’s state. Then at the end of the epoch, you would use
epoch_logger.log_tabular(NameOfQuantity, **options)
to record the desired values.

log_tabular
(key, val=None, with_min_and_max=False, average_only=False)[source]Â¶ Log a value or possibly the mean/std/min/max values of a diagnostic.
Parameters:  key (string) – The name of the diagnostic. If you are logging a
diagnostic whose state has previously been saved with
store
, the key here has to match the key you used there.  val – A value for the diagnostic. If you have previously saved
values for this key via
store
, do not provide aval
here.  with_min_and_max (bool) – If true, log min and max values of the diagnostic over the epoch.
 average_only (bool) – If true, do not log the standard deviation of the diagnostic over the epoch.
 key (string) – The name of the diagnostic. If you are logging a
diagnostic whose state has previously been saved with

Loading Saved Models (PyTorch Only)Â¶
To load an actorcritic model saved by a PyTorch Spinning Up implementation, run:
ac = torch.load('path/to/model.pt')
When you use this method to load an actorcritic model, you can minimally expect it to have an act
method that allows you to sample actions from the policy, given observations:
actions = ac.act(torch.as_tensor(obs, dtype=torch.float32))
Loading Saved Graphs (Tensorflow Only)Â¶

spinup.utils.logx.
restore_tf_graph
(sess, fpath)[source]Â¶ Loads graphs saved by Logger.
Will output a dictionary whose keys and values are from the ‘inputs’ and ‘outputs’ dict you specified with logger.setup_tf_saver().
Parameters:  sess – A Tensorflow session.
 fpath – Filepath to save directory.
Returns: A dictionary mapping from keys to tensors in the computation graph loaded from
fpath
.
When you use this method to restore a graph saved by a Tensorflow Spinning Up implementation, you can minimally expect it to include the following:
Key  Value 

x 
Tensorflow placeholder for state input. 
pi 
Samples an action from the agent, conditioned
on states in
x . 
The relevant value functions for an algorithm are also typically stored. For details of what else gets saved by a given algorithm, see its documentation page.
PlotterÂ¶
See the page on plotting results for documentation of the plotter.
MPI ToolsÂ¶
Table of Contents
Core MPI UtilitiesÂ¶

spinup.utils.mpi_tools.
mpi_fork
(n, bind_to_core=False)[source]Â¶ Relaunches the current script with workers linked by MPI.
Also, terminates the original process that launched it.
Taken almost without modification from the Baselines function of the same name.
Parameters:  n (int) – Number of process to split into.
 bind_to_core (bool) – Bind each MPI process to a core.

spinup.utils.mpi_tools.
mpi_statistics_scalar
(x, with_min_and_max=False)[source]Â¶ Get mean/std and optional min/max of scalar x across MPI processes.
Parameters:  x – An array containing samples of the scalar to produce statistics for.
 with_min_and_max (bool) – If true, return min and max of x in addition to mean and std.
MPI + PyTorch UtilitiesÂ¶
spinup.utils.mpi_pytorch
contains a few tools to make it easy to do dataparallel PyTorch optimization across MPI processes. The two main ingredients are syncing parameters and averaging gradients before they are used by the adaptive optimizer. Also there’s a hacky fix for a problem where the PyTorch instance in each separate process tries to get too many threads, and they start to clobber each other.
The pattern for using these tools looks something like this:
 At the beginning of the training script, call
setup_pytorch_for_mpi()
. (Avoids clobbering problem.)  After you’ve constructed a PyTorch module, call
sync_params(module)
.  Then, during gradient descent, call
mpi_avg_grads
after the backward pass, like so:
optimizer.zero_grad()
loss = compute_loss(module)
loss.backward()
mpi_avg_grads(module) # averages gradient buffers across MPI processes!
optimizer.step()

spinup.utils.mpi_pytorch.
mpi_avg_grads
(module)[source]Â¶ Average contents of gradient buffers across MPI processes.
MPI + Tensorflow UtilitiesÂ¶
The spinup.utils.mpi_tf
contains a a few tools to make it easy to use the AdamOptimizer across many MPI processes. This is a bit hacky—if you’re looking for something more sophisticated and generalpurpose, consider horovod.

class
spinup.utils.mpi_tf.
MpiAdamOptimizer
(**kwargs)[source]Â¶ Adam optimizer that averages gradients across MPI processes.
The compute_gradients method is taken from Baselines MpiAdamOptimizer. For documentation on method arguments, see the Tensorflow docs page for the base AdamOptimizer.
Run UtilsÂ¶
Table of Contents
ExperimentGridÂ¶
Spinning Up ships with a tool called ExperimentGrid for making hyperparameter ablations easier. This is based on (but simpler than) the rllab tool called VariantGenerator.

class
spinup.utils.run_utils.
ExperimentGrid
(name='')[source]Â¶ Tool for running many experiments given hyperparameter ranges.

add
(key, vals, shorthand=None, in_name=False)[source]Â¶ Add a parameter (key) to the grid config, with potential values (vals).
By default, if a shorthand isn’t given, one is automatically generated from the key using the first three letters of each colonseparated term. To disable this behavior, change
DEFAULT_SHORTHAND
in thespinup/user_config.py
file toFalse
.Parameters:  key (string) – Name of parameter.
 vals (value or list of values) – Allowed values of parameter.
 shorthand (string) – Optional, shortened name of parameter. For
example, maybe the parameter
steps_per_epoch
is shortened tosteps
.  in_name (bool) – When constructing variant names, force the inclusion of this parameter into the name.

run
(thunk, num_cpu=1, data_dir=None, datestamp=False)[source]Â¶ Run each variant in the grid with function ‘thunk’.
Note: ‘thunk’ must be either a callable function, or a string. If it is a string, it must be the name of a parameter whose values are all callable functions.
Uses
call_experiment
to actually launch each experiment, and gives each variant a name usingself.variant_name()
.Maintenance note: the args for ExperimentGrid.run should track closely to the args for call_experiment. However,
seed
is omitted because we presume the user may add it as a parameter in the grid.

variant_name
(variant)[source]Â¶ Given a variant (dict of valid param/value pairs), make an exp_name.
A variant’s name is constructed as the grid name (if you’ve given it one), plus param names (or shorthands if available) and values separated by underscores.
Note: if
seed
is a parameter, it is not included in the name.

variants
()[source]Â¶ Makes a list of dicts, where each dict is a valid config in the grid.
There is special handling for variant parameters whose names take the form
'full:param:name'
.The colons are taken to indicate that these parameters should have a nested dict structure. eg, if there are two params,
Key Val 'base:param:a'
1 'base:param:b'
2 the variant dict will have the structure
variant = { base: { param : { a : 1, b : 2 } } }

Calling ExperimentsÂ¶

spinup.utils.run_utils.
call_experiment
(exp_name, thunk, seed=0, num_cpu=1, data_dir=None, datestamp=False, **kwargs)[source]Â¶ Run a function (thunk) with hyperparameters (kwargs), plus configuration.
This wraps a few pieces of functionality which are useful when you want to run many experiments in sequence, including logger configuration and splitting into multiple processes for MPI.
There’s also a SpinningUpspecific convenience added into executing the thunk: if
env_name
is one of the kwargs passed to call_experiment, it’s assumed that the thunk accepts an argument calledenv_fn
, and that theenv_fn
should make a gym environment with the givenenv_name
.The way the experiment is actually executed is slightly complicated: the function is serialized to a string, and then
run_entrypoint.py
is executed in a subprocess call with the serialized string as an argument.run_entrypoint.py
unserializes the function call and executes it. We choose to do it this way—instead of just calling the function directly here—to avoid leaking state between successive experiments.Parameters:  exp_name (string) – Name for experiment.
 thunk (callable) – A python function.
 seed (int) – Seed for random number generators.
 num_cpu (int) – Number of MPI processes to split into. Also accepts ‘auto’, which will set up as many procs as there are cpus on the machine.
 data_dir (string) – Used in configuring the logger, to decide where
to store experiment results. Note: if left as None, data_dir will
default to
DEFAULT_DATA_DIR
fromspinup/user_config.py
.  **kwargs – All kwargs to pass to thunk.

spinup.utils.run_utils.
setup_logger_kwargs
(exp_name, seed=None, data_dir=None, datestamp=False)[source]Â¶ Sets up the output_dir for a logger and returns a dict for logger kwargs.
If no seed is given and datestamp is false,
output_dir = data_dir/exp_name
If a seed is given and datestamp is false,
output_dir = data_dir/exp_name/exp_name_s[seed]
If datestamp is true, amend to
output_dir = data_dir/YYMMDD_exp_name/YYMMDD_HHMMSS_exp_name_s[seed]
You can force datestamp=True by setting
FORCE_DATESTAMP=True
inspinup/user_config.py
.Parameters:  exp_name (string) – Name for experiment.
 seed (int) – Seed for random number generators used by experiment.
 data_dir (string) – Path to folder where results should be saved.
Default is the
DEFAULT_DATA_DIR
inspinup/user_config.py
.  datestamp (bool) – Whether to include a date and timestamp in the name of the save directory.
Returns: logger_kwargs, a dict containing output_dir and exp_name.
AcknowledgementsÂ¶
We gratefully acknowledge the contributions of the many people who helped get this project off of the ground, including people who beta tested the software, gave feedback on the material, improved dependencies of Spinning Up code in service of this release, or otherwise supported the project. Given the number of people who were involved at various points, this list of names may not be exhaustive. (If you think you should have been listed here, please do not hesitate to reach out.)
In no particular order, thank you Alex Ray, Amanda Askell, Ben Garfinkel, Christy Dennison, Coline Devin, Daniel Zeigler, Dylan HadfieldMenell, Ge Yang, Greg Khan, Jack Clark, Jonas Rothfuss, Larissa Schiavo, Leandro Castelao, Lilian Weng, Maddie Hall, Matthias Plappert, Miles Brundage, Peter Zokhov, and Pieter Abbeel.
We are also grateful to Pieter Abbeel’s group at Berkeley, and the Center for HumanCompatible AI, for giving feedback on presentations about Spinning Up.
About the AuthorÂ¶
Spinning Up in Deep RL was primarily developed by Josh Achiam, a research scientist on the OpenAI Safety Team and PhD student at UC Berkeley advised by Pieter Abbeel. Josh studies topics related to safety in deep reinforcement learning, and has previously published work on safe exploration.