Teaching a pixel robot to stack crates

A 3,268-parameter PPO policy, trained in a numpy sandbox, running in your browser at 60 Hz.

by aadi shah·August 2026

The live policy needs a wider screen than this one, so the animation is hidden on mobile — open this page on a laptop to watch the arm run. The write-up below is all here either way.

The arm above is not keyframed. Each frame an observation is read off the simulator, pushed through a small MLP, and the four outputs become base velocity, two joint velocities, and a gripper command. The behavior is whatever maximized return over 32M steps of experience.

01The MDP

Reinforcement learning problems are written down as a Markov decision process, which is a formal way of saying: a loop, running one tick at a time. The agent sees the state — a list of numbers describing the world right now. It picks an action. The world advances one tick and hands back the next state and a scalar reward. Then it happens again.

The Markov part is the constraint that makes this tractable: the state has to contain everything relevant about the future, so the agent can decide from the current state alone and never needs to remember how it got there. Learning means finding a policy, a function from states to actions:

The subscript is the thing being learned — here $\theta$ is the weights of a small neural network. It is written as a distribution over actions rather than a single action because during training the policy samples, which is how it explores; at deployment it takes the distribution's mean. The spaces $\mathcal{S}$ and $\mathcal{A}$ are both continuous here:

The world itself is the 480×60 pixel strip above: a wheeled base at $x_b$, a two-link arm with segments of 16 and 14 px, a gripper, and two crate columns 390 px apart.

02Why PPO

There is no dataset here. Nobody knows the right action for a given state, so there is nothing to imitate — the only way to find out whether an action was any good is to take it and see what comes back. That makes this online learning: the algorithm generates its own training data by acting, and what it learns changes what it collects next. The 32M steps behind this policy are not 32M examples someone labeled; they are 32M ticks it played out itself.

The quantity being maximized is the expected return $J(\theta )={\mathbb{E}}_{\tau \sim {\pi }_{\theta }}\left[{\sum }_t{\gamma }^t{r}_t\right]$, the average total reward the policy collects over a trajectory. It cannot be differentiated directly — the reward comes from a simulator, not from a formula in $\theta$. The policy gradient theorem gets around that by moving the derivative onto the one part that is a differentiable function of $\theta$, the policy's own log-probability:

Which reads: push up the probability of actions that turned out better than expected, push down the ones that turned out worse. “Than expected” needs a definition, and it comes from two standard quantities. The value of a state is the return the policy expects to collect from there onward; the action-value is the same thing if you commit to one action first. Their difference is the advantage:

So $A>0$ means the action did better than the policy's own average from that state, and $A<0$ means it did worse. Neither $V$ nor $Q$ is known, so $V$ is learned by a second network — the critic, $V_{\varphi }$ — and ${\hat{A}}_t$ is estimated from it and the rewards actually observed. Using the value as a subtracted baseline this way does not bias the gradient; it only reduces its variance.

Estimate the outer expectation by averaging over a batch of collected steps and you have the vanilla policy gradient, REINFORCE [5] with a baseline. It is unbiased, it is about five lines of code, and on its own it is almost unusable: the estimate is so noisy that useful steps are small, and large ones are a coin flip.

The reason you cannot simply take bigger steps is that the expectation is over $\tau \sim {\pi }_{\theta }$ — trajectories drawn from the policy as it currently is. The gradient is only valid at the parameters that collected the data, which makes the method on-policy: each batch describes one policy, and the moment you update, it is stale. Supervised data keeps; this does not. So a step that overshoots leaves you somewhere worse holding a batch that no longer describes where you are, and the usual remedy — reuse the data to correct course — is exactly what is unavailable. In supervised learning a bad step costs an epoch. Here it can cost the run.

TRPO [2] answers this with a hard constraint on the KL divergence between old and new policy, solved with second-order methods. PPO [1] gets most of the same effect from one first-order trick, and it buys something else along the way: it is safe to take several gradient steps on the same batch, which is what makes an on-policy method affordable at all.

Look at how much more or less likely an action has become since the data was collected:

At the moment of collection $r_t=1$. As the update proceeds it drifts: above 1 means the action became more likely, below 1 less. This ratio is the thing PPO polices, and it does so by making the objective stop rewarding drift past $1\pm ϵ$:

Two cases, and they are worth walking through separately.

Good action (${\hat{A}}_t>0$). Raising its probability raises the objective, so the gradient wants to keep pushing. But past $r_t=1+ϵ$ the clipped branch is the smaller of the two, the $\min$ selects it, and it is a constant in $\theta$ — flat, no gradient. One good sample can therefore move the policy by at most a bounded amount, no matter how large its advantage.

Bad action (${\hat{A}}_t<0$). Now the objective improves by making the action less likely, and the ceiling sits at $r_t=1-ϵ$. Below that the term flattens, so a single bad sample cannot drive its own probability to zero.

The asymmetry that makes this work is that the un-clipped branch is still chosen whenever it is smaller. If an update has already overshot and made things worse, the objective still registers it, so the policy can walk back rather than being pinned at the boundary. Note also what PPO is not doing: there is no constraint, no penalty, and nothing forbids $r_t$ from leaving the interval. Leaving simply stops paying.

That leaves ${\hat{A}}_t$, which has to be estimated from a single trajectory. The one-step version is

Low variance, but it inherits every error in $V$. Summing rewards to the end of the episode instead is unbiased but noisy, since one lucky episode drags the estimate. GAE [3] interpolates between them with a single knob:

$\lambda =0$ recovers the one-step estimate, $\lambda =1$ the full Monte Carlo return, and values in between blend them geometrically. $\lambda =0.95$ is what this run uses: enough smoothing to survive 600-step episodes, enough reach to carry the terminal placement bonus back through the drive that earned it.

The loss that actually gets minimized has three parts:

The value term trains the critic by regressing $V_{\varphi }$ onto observed returns; it is the same critic whose predictions become ${\hat{A}}_t$. The entropy term pays the policy to stay random, which early on is what lets it stumble into a grasp at all. It anneals from $c_e=4\times 10^{-3}$ to $8\times 10^{-4}$ as the behavior forms.

The browser runs the mean action ${\mu }_{\theta }(s_t)$ and does not sample, so the Gaussian is training machinery only.

03Reward

PPO maximizes whatever you write down. Not what you meant — what you wrote. Most of the work here was not tuning the algorithm; it was discovering, repeatedly, that the reward I had written could be satisfied by something other than stacking crates.

The obvious reward — a point per crate, nothing otherwise — never gets off the ground. A random policy would have to drive over, lower onto a crate, close, drive back, and open at the right height entirely by accident before it saw any feedback, and until it sees feedback there is no gradient to follow. So the reward pays for progressevery step instead:

$d_t$ is the distance from the gripper to whatever it currently wants — the crate if empty, the drop slot if carrying — and that term does most of the work. Everything else is a patch over a way the policy found to score without doing the task:

The windmill is the one worth unpacking, because the penalty was already there and simply did not work. The network does not output joint speed directly; it outputs $u_t$, squashed into range by $\dot{q}=\tanh (u){\dot{q}}_{\max }$. Penalize the speed and the chain rule hands you a factor of $1-{\tanh }^2(u)$. Early training rewards turning the shoulder as fast as it will go, which pushes $u$ to around $4.2$, where that factor is $8\times 10^{-4}$. Raising the coefficient six-fold changed nothing, because six times almost zero is almost zero.

Penalizing $u$ itself fixes it: that gradient grows as the output saturates instead of vanishing, pulling the command back into the range where the joint is steerable. Shoulder travel fell from 37.9 radians an episode to 13, and time at full deflection from 71% of steps to zero.

04Training

The full task from scratch does not converge in any reasonable budget, so it is split into four stages, each warm-starting the next:

Stage 3 was added last and mattered most. Grasping had a stage from the start; placing did not, so the only placements in the data were ones the policy stumbled into during the whole task — and in the ferry direction it happened to be worse at, it never stumbled into any. A skill with no successful examples never starts. The result was 100% carrying crates left to right and 0% carrying them back, which is exactly what an averaged success rate hides. Training placement directly, from both sides, took the combined figure to 100%.

Hyperparameters:

05Where to learn this

Roughly the order I would take them in:

• Reinforcement Learning: An Introduction (Sutton & Barto) for the foundations: MDPs, value functions, policy gradients.
• CS224R (Chelsea Finn, Stanford) for modern deep RL, with a more practical focus.
• Deep RL Course (Hugging Face) for learning by doing: you train agents as you go.
• Spinning Up (OpenAI) as a reference: short derivations next to readable single-file implementations.

06References

1. [1]Schulman, Wolski, Dhariwal, Radford, Klimov. Proximal Policy Optimization Algorithms. OpenAI, arXiv:1707.06347, 2017.
2. [2]Schulman, Levine, Moritz, Jordan, Abbeel. Trust Region Policy Optimization. arXiv:1502.05477, 2015.
3. [3]Schulman, Moritz, Levine, Jordan, Abbeel. High-Dimensional Continuous Control Using Generalized Advantage Estimation. arXiv:1506.02438, 2015.
4. [4]Kingma, Ba. Adam: A Method for Stochastic Optimization. arXiv:1412.6980, 2014.
5. [5]Williams. Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning. Machine Learning 8, 1992.