Cheat sheets
Every week’s key equations on one page
Week 1 · Introduction to Robot Learning
| Name | Formula | Where |
|---|---|---|
| Supervised learning | \(\min_\theta \frac{1}{N}\sum_i \ell(f_\theta(x_i), y_i)\), i.i.d. data | eq. |
| Robot loop | \(a_t \sim \pi(\cdot \mid o_t)\), \(s_{t+1} \sim p(\cdot \mid s_t, a_t)\) | eq. |
| Open vs closed loop | \(\text{plan}_t(\hat{s}_0)\) vs \(\pi(\hat{s}_t)\) | eq. |
| Tokenization | \(k = \lfloor (a - a_{\min})/\Delta \rfloor\), \(\lvert a - \hat{a} \rvert \le \Delta/2\) | eq. |
| Sequence model | \(\prod_j p(k_j \mid \text{image}, \text{text}, k_{<j})\) | eq. |
Week 2 · Robot Control & MDPs
| Name | Formula | Where |
|---|---|---|
| Homogeneous transform | \(\mathbf{p}' = T\, \mathbf{p}\) | eq. |
| Grübler–Kutzbach | \(m(N{-}1{-}J) + \sum f_i\) | eq. |
| Kinematics | \(x = f(q)\) | eq. |
| Jacobian IK step | \(q \mathrel{-}= \alpha J^\top e\) | eq. |
| Quintic scaling | \(10s^3 - 15s^4 + 6s^5\) | eq. |
| PID ★ | \(K_p e + K_i \textstyle\sum e \Delta t + K_d \dot e\) | eq. |
| Markov property | \(p(s_{t+1} \mid s_t, a_t)\) | eq. |
| Trajectory probability | \(p(s_0) \prod_t \pi P\) | eq. |
| RL objective ★ | \(\max_\pi \mathbb{E}[\sum_t \gamma^t r_t]\) | eq. |
| Near vs far | \((R_{\text{near}}/R_{\text{far}})^{1/\Delta d}\) | eq. |
| Random search | \(\theta \mathrel{+}= \frac{\alpha}{N} \sum_k \Delta r_k \delta_k\) | eq. |
Week 3 · Imitation Learning
| Name | Formula | Where |
|---|---|---|
| BC loss | \(\min_\theta \sum_{\mathcal{D}} \lVert \pi_\theta(s) - a^* \rVert^2\) | eq. |
| MSE minimizer | \(c^\star = \mathbb{E}[a^* \mid s]\) | eq. |
| BC compounding error ★ | \(\mathbb{E}[\sum_t c] = O(\varepsilon T^2)\) | eq. |
| DAgger data rule, cost \(O(\varepsilon T)\) | \(\mathcal{D} \leftarrow \mathcal{D} \cup \{(s, \pi^*(s))\}\), \(s \sim p_{\pi_\theta}\) | eq. |
| Mixture of Gaussians | \(\sum_k w_k \mathcal{N}(a; \mu_k, \Sigma_k)\) | eq. |
| Autoregressive actions | \(\prod_d p(a_{t,d} \mid s_t, a_{t,<d})\) | eq. |
| ELBO ★ | \(\mathbb{E}_q[\log p(\tau \mid z, c)] - \mathrm{KL}(q \Vert p)\) | eq. |
Week 4 · Reinforcement Learning I
| Name | Formula | Where |
|---|---|---|
| Smoothed reward | \(2\Phi\big(\frac{c - \mu}{\sigma}\big) - 1\) | eq. |
| Value | \(V^\pi(s) = \mathbb{E}[\sum_t \gamma^t r_t \mid s]\) | eq. |
| Bellman ★ | \(V = r + \gamma P V\) | eq. |
| Bellman optimality ★ | \(V^* = \max_a [r + \gamma P V^*]\) | eq. |
| Value iteration | \(V_{k+1} = \max_a [r + \gamma P V_k]\) | eq. |
| Policy iteration | evaluate, then greedy | eq. |
| Optimal Q | \(\pi^*(s) = \arg\max_a Q^*(s, a)\) | eq. |
| Q-learning ★ | \(Q \mathrel{+}= \alpha(r + \gamma \max Q' - Q)\) | eq. |
| SARSA | \(Q \mathrel{+}= \alpha(r + \gamma Q(s', a') - Q)\) | eq. |
| DQN loss | \((r + \gamma \max Q_{\theta^-} - Q_\theta)^2\) | eq. |
| Double target | \(Q_{\theta^-}(s', \arg\max Q_\theta)\) | eq. |
| DDPG actor | \(\nabla_a Q \cdot \nabla_\phi \mu_\phi\) | eq. |
Week 5 · Reinforcement Learning II
| Name | Formula | Where |
|---|---|---|
| Objective | \(J(\theta) = \mathbb{E}_{\tau \sim p_\theta}[R(\tau)]\) | eq. |
| Log-derivative trick | \(\nabla p = p\, \nabla \log p\) | eq. |
| Policy gradient ★ | \(\mathbb{E}[\sum_t \nabla \log \pi\, R(\tau)]\) | eq. |
| Reward-to-go, baseline | \(\sum_{t' \ge t} r_{t'} - b\) | eq. |
| Advantage | \(A = Q - V\) | eq. |
| Importance sampling | \(\mathbb{E}_p[f] = \mathbb{E}_q[\frac{p}{q} f]\) | eq. |
| Off-policy gradient | per-step ratio \(\times \nabla \log \pi \times A\) | eq. |
| Entropy bonus | \(L + \beta \mathcal{H}\) | eq. |
| TRPO ★ | max \(\mathbb{E}[\rho A]\) s.t. KL \(\le \delta\) | eq. |
| PPO clip ★ | \(\min(\rho A, \operatorname{clip}(\rho)A)\) | eq. |
| Full PPO | clip − value loss + entropy | eq. |
| SAC | \(\alpha \log \pi - Q\); soft TD target | eq. |
| Reparameterization | \(a = \mu + \sigma \varepsilon\) | eq. |
Week 6 · Generative Models
| Name | Formula | Where |
|---|---|---|
| VAE bound ★ | \(\mathbb{E}_q \log p(x \mid z) - \mathrm{KL}\) | eq. |
| Reparam. trick | \(z = \mu + \sigma \odot \varepsilon\) | eq. |
| VQ codes | \(z_e + \mathrm{sg}[e_k - z_e]\) | eq. |
| Forward step | \(\sqrt{\alpha_i}\,x_{i-1} + \sqrt{\beta_i}\,\varepsilon\) | eq. |
| Jump to step \(i\) | \(\sqrt{\bar\alpha_i}\,x_0 + \sqrt{1-\bar\alpha_i}\,\varepsilon\) | eq. |
| DDPM loss ★ | \(\lVert \varepsilon - \varepsilon_\theta \rVert^2\) | eq. |
| DDIM | estimate \(\hat x_0\), re-noise to \(i'\) | eq. |
| Guidance | \(\varepsilon_\varnothing + w\,\Delta\varepsilon\) | eq. |
| Flow matching | \(\lVert v_\theta - (x_0 - \varepsilon) \rVert^2\) | eq. |
Week 7 · Sequence Modeling and Transformers
| Name | Formula | Where |
|---|---|---|
| Chain rule ★ | \(p(x_{1:T}) = \prod_i p(x_i \mid x_{1:i-1})\) | eq. |
| RNN | \(h_t = f(h_{t-1}, x_t)\) | eq. |
| Attention ★ | \(\operatorname{softmax}(QK^\top/\sqrt{d})\,V\) | eq. |
| Causal mask | \(s_{ij} = -\infty\) for \(j > i\) | eq. |
| Positional encodings | \(e_i + \mathrm{PE}(i)\); \(s_{ij} + b(i - j)\) | eq. |
| RoPE | \((R_i q)^\top (R_j k) = q^\top R_{j-i} k\) | eq. |
| Teacher forcing | \(-\frac{1}{T}\sum_t \log p(x_t \mid x_{<t})\) | eq. |
| Chinchilla ★ | \(C \approx 6ND\), \(D^* \approx 20 N^*\) | eq. |
| CLIP | symmetric softmax over \(s_{ij}/\tau\) | eq. |
| Flamingo gate | \(x + \tanh(\alpha)\,\text{CrossAttn}\) | eq. |
| Action bins | quantile-normalize, 256 bins | eq. |
| Chunk, ensemble | \(\pi(a_{t:t+k} \mid o_t)\), \(w_i = e^{-mi}\) | eq. |
| DCT (FAST) | \(c_j = \nu_j \sum_n u_n \cos(\cdot)\) | eq. |
Week 8 · World Models
| Name | Formula | Where |
|---|---|---|
| Policy vs world model | \(\pi(a \mid s, g)\) vs \(p(s' \mid s, a)\) | eq. |
| World model ★ | \(p(s_{t+1} \mid s_t, a_t, h_t)\) | eq. |
| MPC | argmin of imagined cost, execute first step, replan | eq. |
| Compounding bound | \(e_H \le \varepsilon (L^H - 1)/(L - 1)\) | eq. |
| World Models (V-M-C) | VAE, MDN-RNN, linear controller | eq. |
| RSSM | \(h_t\) deterministic; \(q(z \mid h, o)\), \(p(z \mid h)\), KL | eq. |
| Dreamer | maximize imagined return + critic | eq. |
| Video tokens | \(\frac{HW}{p^2} f T / (c_s^2 c_t)\) | eq. |
| WAM, VAM | joint \(p(o, a)\); or \(\hat o\) then IDM | eq. |
| JEPA | \(\|\mathrm{pred}(z_t, a_t) - \mathrm{sg}[z_{t+1}]\|^2\) | eq. |
Week 9 · Generalist Robot Policies
| Name | Formula | Where |
|---|---|---|
| Goal-conditioned BC | \(-\log \pi(a_{t:t+H-1} \mid o_{t-h:t}, g)\) | eq. |
| Multicontext imitation | one policy, encoder \(f_k\) per kind of task description | eq. |
| Masked multi-robot loss | \(\sum_j m_{ij} (\hat a_{ij} - a_{ij})^2\) | eq. |
| Gato’s loss | next-token loss on text and action tokens only | eq. |
| Mu-law ★ | \(\operatorname{sgn}(x) \log(1 + \mu|x|) / \log(1 + M\mu)\) | eq. |
| Padding waste | \(1 - \sum L_i / (B \max L_i)\) | eq. |
| Packing | cu_seqlens, block-diagonal mask, positions reset | eq. |
| VLA recipe | next-token loss + flow loss through \(\mathrm{sg}[h_\theta]\) | eq. |
| Advantage conditioning | \(I_t = \mathbb{1}[A > \epsilon_\ell]\); clone with and without \(I\) | eq. |
| MMRV | mean worst misordered real gap | eq. |
| SysID | least-squares gain, residual noise | eq. |
| Success-rate SE | \(\sqrt{\hat p (1 - \hat p) / n}\) | eq. |
Week 10 · Embodied Reasoning and Test-time Scaling
| Name | Formula | Where |
|---|---|---|
| Embodied CoT | \(-\log p(c \mid o, \ell) - \log p(a \mid o, \ell, c)\) | eq. |
| Reasoning dropout | as ECoT, with \(m \cdot c\), \(m \sim\) Bernoulli\((1 - p_{\text{drop}})\) | eq. |
| Best-of-N | \(1 - (1 - p)^N\) | eq. |
| Learned verifier | \(\arg\max_i V(o, a_i)\); capped by \(V\)’s own errors | eq. |
| Elo ★ | \(1/(1 + 10^{-\Delta/400})\); 120 Elo ≈ one doubling | eq. |
| Reasoning as depth | sequential steps \(\approx L (T + 1)\) | eq. |
| Group advantage | \((r_i - \text{mean}\, r)/\text{std}\, r\) | eq. |
| GRPO ★ | clipped ratio × \(\hat A\), minus \(\beta\) KL to \(\pi_{\text{ref}}\) | eq. |
| Learning signal | \(1 - (1 - p)^G\) | eq. |
Week 11 · Frontiers & Open Problems
| Name | Formula | Where |
|---|---|---|
| Tabular cloning | \(N(s, a) / \sum_{a'} N(s, a')\) | eq. |
| Hindsight relabelling | \(-\sum \log \pi(a_t \mid s_t, g = s_{t'})\) | eq. |
| Bellman on the data ★ | \(V(s) = \min_{(s,a,s') \in \mathcal{D}} [1 + V(s')]\) | eq. |
| Expert share | \(N_e / (N_e + k M)\) | eq. |
| Ensemble disagreement | \(\sum_j \operatorname{std}_m \pi_m(o)_j\) | eq. |
| Acting selectively | act if \(u \le \tau\); coverage, selective success | eq. |
| CLIP loss ★ | softmax over rows and columns of \(t\, X Y^\top\) | eq. |
| SigLIP loss | \(-\sum \log \sigma(z_{ij}(t\, x_i \cdot y_j + b)) / n\) | eq. |
| Hard negative | \(1 - (1 - f)^{n - 1}\) | eq. |
| Video memory | seconds × fps × tokens per frame | eq. |
Week 12 · Guest Lectures
No cheat sheet yet.