Teaching a pixel robot to stack crates
How to use reinforcement learning to teach an old robot new tricks.
by aadi shah·August 2026·10 min read
For a fun addition to my website, I decided to make a small demo of a robot arm learning how to pick up and stack crates. This runs using a (very) small neural network that takes in each simulator state and outputs the motion commands for the arm, base, and gripper. After 32 million steps of learning through trial-and-error, we end up with the policy deployed above.
01Problem formulation
A task like this is usually framed as a Markov decision process. At each step the agent sees a state, a compact summary of what's going on, and picks an action. The environment moves forward and hands back a reward, a scalar quantity describing the utility of the action.
Reinforcement learning is about learning a policy from that feedback by trial and error. What counts is the total reward over the whole episode, so an action can be worth taking even if it pays off little right away.
The Markov property states that the probability of future states depends only on our current state and action, and not on any previous ones. Using this formulation, we try to learn a policy, a mapping from our state space to our action space, . Our policy can be either stochastic, outputting a distribution over actions, or deterministic, resulting in a fixed action:
Ours is a stochastic policy , where is the weights of a small neural network. During training the policy samples from that distribution so it keeps trying new things. In the browser it just takes the mean. Both states and actions are continuous:
02Learning with PPO
Now that we've formulated our problem and approach, how do we actually find a good policy? There's no labeled data telling us the right action in each state, so the policy has to learn from its own experience: act in the simulator, observe the reward, and update. This makes training online, since the data comes from the current policy, and every update changes what the next rollout looks like.
Policy gradients:
We want to maximize the expected return . Here is one rollout and discounts rewards that arrive later. The catch is that we can't backprop through the simulator. The policy gradient theorem gets around this by rewriting the gradient in terms of the policy's log-probability, which we can differentiate with respect to . This results from the log-derivative trick [8], an awesome sleight of hand that leads to REINFORCE [5], or the vanilla policy gradient approach. REINFORCE samples full rollouts from the current policy, then increases the log-probability of each action in proportion to the total discounted reward of its rollout:
We can't compute this expectation exactly, so we estimate it by averaging over sampled rollouts, then take a gradient ascent step with learning rate :
Repeating these steps gives the full REINFORCE algorithm:
In practice, we don't compute each separately, which would take backward passes. Instead, we optimize a surrogate objective whose gradient is exactly our estimate, and let autograd differentiate through it in a single backward pass:
One simple improvement is to subtract a baseline , such as the average reward across rollouts, from each rollout's total reward. This doesn't bias the gradient, since the expected gradient of the log-probability is zero, but it can greatly reduce its variance.
Actor-critic methods:
Instead of a single constant baseline, actor-critic methods learn what's good and bad in each state. The policy (the actor) picks actions, and a second network (the critic) judges whether each one went better or worse than expected. The update then makes the better actions more likely and the worse ones less likely. We define the value of a state as the return the policy expects from there on, and we define the action-value as the same thing if you commit to one action first. The advantage is the gap between them:
means the action beat the policy's own average from that state, and means it fell short. We don't know or , so a second network, the critic , learns , and is estimated from the critic plus the rewards we actually saw. We can then plug into our surrogate objective in place of the summed rewards, giving .
Advantage estimation:
Since we can't directly observe the advantage, we estimate it at each step, , using the rewards that followed and the critic's value predictions. The simplest estimate is the temporal difference (TD) residual, which only looks one step ahead:
This estimate has low variance, but it's biased, since it relies on the critic's value estimate being accurate. At the other extreme, a Monte Carlo estimate sums the actual rewards until the end of the episode, which is unbiased but has high variance. Generalized Advantage Estimation (GAE) [3] interpolates between the two using a parameter :
recovers the one-step estimate, the Monte Carlo return minus the baseline, and anything in between is a geometric blend. I used , which keeps variance manageable without leaning too hard on a critic that starts out pretty bad.
Clipped objective:
Now that we have an advantage estimate for every step, the remaining question is how large an update to take. Our surrogate objective is only valid for the policy that collected the rollouts, since . This is what makes the method on-policy. After a single update, the batch describes an older policy, so reusing it without correction makes our gradient estimate less accurate. A large update can therefore make the policy worse and also leave us with data that no longer describes it.
Trust Region Policy Optimization (TRPO) [2] addresses this by constraining the KL divergence, a measure of how different two distributions are, between the old and new policy. Enforcing this constraint requires second-order optimization, which uses curvature information and is expensive to compute. Proximal Policy Optimization (PPO) [1] instead uses a clipped first-order objective. This lets us take several gradient steps on the same batch without rewarding any one action's probability for changing too much in a single update.
To do this, PPO tracks the ratio between the new and old policy's probability for each sampled action:
When the data is collected, . During the update, means the action became more likely, and means it became less likely. PPO clips this ratio to the interval :
Positive advantage (). Increasing the action's probability improves the objective, but only until . Beyond that, the selects the clipped term, which is constant in , so that sample stops contributing gradient.
Negative advantage (). Decreasing the action's probability improves the objective, but only until , so a single bad sample can't push its own probability to zero.
Clipping is not a hard constraint. The still selects the unclipped term whenever it's smaller, so if an update moves the ratio in the wrong direction, the gradient pulls it back. Clipping only removes the incentive to keep moving in a direction that is already helping.
Total loss:
Putting it together, the loss has three parts:
The value term trains to predict the returns we actually observed, which is what feeds . The entropy term keeps the action variance from collapsing too early, so the policy keeps exploring. I annealed its coefficient from to over training.
In the browser the policy just outputs its mean action and never samples. The Gaussian only matters during training.
03Reward
One of the most important parts of reinforcement learning is properly defining your reward. Oftentimes, your policy learns to cheat your defined reward function (whether that be parametric or non-parametric) in order to get high rewards without learning the intended behavior. We call this “reward hacking” [7]. Defining the reward for this setup was a bit hacky, but here's a high-level overview of some of the challenges:
Sparse and shaped rewards:
My first attempt used a sparse reward: one point for placing a crate and zero otherwise. This failed, since a random policy would have to drive over, lower onto a crate, close the gripper, drive back, and open at the right height, all by chance, before ever receiving a nonzero reward. Until then, there is no learning signal.
To fix this, I switched to a dense (shaped) reward that pays out for making progress toward the current target. The final version has the following terms:
Here is the distance from the gripper to its current target, which is the crate when the gripper is empty and the drop slot when it's carrying one. This progress term provides most of the learning signal.
Failures and fixes:
Every other term was added in response to a specific way the policy hacked the reward. The left column describes the behavior, and the right column describes the fix.
04Training
Learning the whole task from scratch did not work within my training budget, so I used curriculum learning: I split the task into four stages, each initialized from the previous stage's checkpoint:
Curriculum stages:
Configuration:
05Further reading
If you want to explore more, check these out:
- Reinforcement Learning: An Introduction (Sutton & Barto) for the foundational theory.
- CS224R (Chelsea Finn, Stanford) for modern deep RL, with a more practical focus.
- Deep RL Course (Hugging Face). I haven't taken this one personally.
- Spinning Up (OpenAI) for a quick reference guide and intuitive explanations.
Thanks for reading!
06References
- [1]Schulman, Wolski, Dhariwal, Radford, Klimov. Proximal Policy Optimization Algorithms. OpenAI, arXiv:1707.06347, 2017.
- [2]Schulman, Levine, Moritz, Jordan, Abbeel. Trust Region Policy Optimization. arXiv:1502.05477, 2015.
- [3]Schulman, Moritz, Levine, Jordan, Abbeel. High-Dimensional Continuous Control Using Generalized Advantage Estimation. arXiv:1506.02438, 2015.
- [4]Kingma, Ba. Adam: A Method for Stochastic Optimization. arXiv:1412.6980, 2014.
- [5]Williams. Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning. Machine Learning 8, 1992.
- [6]Ng, Harada, Russell. Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping. ICML, 1999.
- [7]Weng. Reward Hacking in Reinforcement Learning. Lil'Log, 2024.
- [8]Achiam. Spinning Up in Deep RL, Part 3: Intro to Policy Optimization. OpenAI, 2018.