pixels to actions
This post is about something that doesn't sound significant when you first learn about it, but once u fully realise how it works it makes u go "wait thats actaully inssaneeee".
diffusion policies: how the same core idea behind generating images ended up being one of the best ways to teach a robotic manipulator how to pick up a cup (and other tasks :D)
But what even is a diffusion model?
Its basically training a neural network to generate meaningful images starting from random noise.
how it works: First take an image. Add noise to it, keep adding noise until it looks like pure static. Now train a neural network to reverse that process i.e given any noisy image at step $t$, predict what it looked like at step $t-1$.
the nice thing is u dont have to add noise one step at a time, theres a closed form that jumps straight to any noise level:
$x_0$ is the clean image, $\epsilon$ is pure gaussian noise, and $\bar{\alpha}_t$ is just a knob that decides how much of the original survives at step $t$. crank $t$ up and the first term vanishes, so u just get pure noise.
do this enough times and you have taught the network to "denoise". At inference you start from pure random noise and iteratively denoise your way to a clean image.
thats it.
hence the magic is'nt just one step, its the fact that learning to denoise forces the network to understand the structure of the data. To remove noise for any scene, for example lets say a face, you have to know what faces look like.
and the actual training objective is almost stupidly simple, u just predict the noise that was added:
$\epsilon_\theta$ is the network. u give it the noisy image $x_t$ and the step $t$, it guesses the noise $\epsilon$, and u just minimise the squared error. thats the whole loss.
the denoising objective is secretly just a really powerful way to model a distribution.
okay but what does any of this have to do with robots?
heres where it gets fun.
when u do imitation learning, basically teaching a robot by showing it demonstrations, the whole thing boils down to one question: given what the robot is seeing right now, what action should it take?
the obvious move is to just train a network to map observation → action directly. collect a 1000 demos, regress on them, done.
and this works! ... until it doesn't.
the catch is that demonstrations are multimodal. imagine showing 10 people the same cup on a table and asking them to pick it up. they'll all come at it from different angles, different wrist rotations, different speeds. and heres the thing, all of them are correct. there is'nt one right answer.
a normal regression network HATES this. when it sees the same input mapping to a bunch of different outputs it just averages them. so u end up with an action thats a blurry mix of every valid option, which in robot land usually means the arm reaches halfway to the cup and then just ... freezes. confused.
this is called mode averaging and its a proper fundamental problem, not just some tuning issue u can hack around.
the diffusion policy insight
so chi et al. (2023) asked a really simple question. what if instead of predicting the action directly, we model the distribution over actions, the exact same way diffusion models the distribution over images?
so instead of
observation → action
u get
observation + noisy_action → slightly_less_noisy_action (repeat T times)
and the loss is literally the same one from before, u just swap the image for an action chunk and add the observation as conditioning:
$A_t^k$ is the noisy action chunk at denoising step $k$, $O_t$ is what the robot sees, and the network predicts the noise on the action. same denoising trick, the observation just steers it.
at inference u start from pure noise in action space and denoise your way to a clean action, conditioned on what the robot is currently seeing.
and this is the part that made me go "ohhh thats clever". why does it fix mode averaging? because a diffusion model doesn't average the modes, it SAMPLES from them. if there are 3 valid ways to grab the cup the model can represent all 3, and at inference it just picks one depending on the noise it started from.
u basically went from a network that spits out one answer to a network that understands the entire landscape of reasonable answers. huge.
it doesn't predict just one action
small thing thats easy to miss, diffusion policy doesn't predict a single action. it predicts an action chunk, a little trajectory of the next T actions (like the next 16 timesteps of joint positions).
why does that matter?
- temporal consistency, the robot is'nt making a fresh independent decision every 10th of a second, its committing to a short plan. way less jittery.
- it captures structure, manipulation tasks have clear phases (reach → grasp → lift). a single step prediction cant really capture that, a trajectory can.
and the denoising happens over the whole chunk at once so the network learns how consecutive actions relate to each other as part of the process. neat.
okay but what about the actual network
there are two main backbones people use for the denoiser :
1D temporal UNet, treats the action sequence like a 1d signal and runs convolutions across time. great when the horizon is longer and temporal structure actually matters. kinda overkill for very short horizons (≤8 steps) where the receptive field advantage just disappears.
transformer, treats each timestep as a token and does attention over the sequence. more flexible, better at long range dependencies, but heavier on compute.
the observation (what the robot sees) gets injected either through FiLM conditioning or by just prepending observation tokens, depending on which backbone u pick.
okay but does it actually work
yeah. surprisingly well.
on the robomimic benchmarks even with a simple 3 layer mlp denoiser :
- lift : ~98%
- can : ~96%
- square : ~82%
square is the hard one because it needs precise peg insertion, which is exactly where multimodality hurts normal behaviour cloning the most. diffusion handles it way better cause it can actually represent the uncertainty in the approach angle instead of averaging it into mush.
the whole thing in one line
if u forget everything else from this post remember this :
diffusion works for images because images live on a messy multimodal distribution. robot actions are literally the same. treat action prediction as a denoising problem and u get a model that can represent that messiness instead of collapsing it.
thats the "great intuition" honestly, its not some clever math hack, its just realising the problem structure is the same. images and robot actions are both high dimensional, structured and multimodal. the tool that works for one just ends up working for the other.
so whats next
diffusion policy is genuinely great but its not the end of the road. some directions people are pushing rn :
- consistency models / flow matching: faster inference for the same quality. ddim already took sampling from 100 steps down to ~10, flow matching pushes it even further.
- steering diffusion with RL: instead of just copying demos, using rl to nudge the denoising towards higher reward actions. the noise vectors basically become a latent action space for an rl policy to optimise over. (this ones my favourite ngl)
- multimodal conditioning: feeding in force, tactile and language signals as extra conditioning (ForceVLA, pi0, etc).
but the core idea, model actions as a distribution not a single point, thats here to stay.
thanks for reading :D if u wanna nerd out about any of this hit me up.