Glossary
Vocabulary and notation, grouped by week
Symbols follow the lecture slides. Where a week’s page uses a symbol, it explains it there too.
Week 1 · Introduction to Robot Learning
| Symbol or term | Meaning |
|---|---|
| \(s_t\), \(o_t\), \(a_t\) | State of the world, the robot’s observation of it, and its action, at time step \(t\) |
| \(\hat{s}_t\) | The robot’s estimate of the state, from its sensors |
| \(\pi\) | Policy: the rule that picks an action from what the robot observes |
| \(f_\theta\), \(\ell\) | A model with parameters \(\theta\), and the loss it is trained to minimize |
| i.i.d. | Independent and identically distributed data, the usual assumption of supervised learning |
| \(B\), \(\Delta\) | Number of bins per action dimension, and the bin width |
| \(k\) | An action token: the index of a bin, from 0 to \(B-1\) |
| \(\hat{a}\) | The bin centre a token stands for, the action the robot executes |
| Sense-plan-act | Perceive once, plan a complete path, execute it without looking again |
| Open loop, closed loop | Executing a fixed plan, versus choosing each action from the latest observation |
| Vision-language-action model (VLA) | A vision-language model fine-tuned to output robot actions as tokens |
Week 2 · Robot Control & MDPs
| Symbol or term | Meaning |
|---|---|
| SE(2), SE(3) | Rigid-body poses in the plane (3 numbers) and in space (6 numbers) |
| \(R\), \(\mathbf{t}\), \(T\) | A rotation matrix, a translation vector, and the homogeneous transform built from them |
| \(q\) | Joint configuration: one angle (or length) per joint |
| \(x = f(q)\) | Forward kinematics: the end-effector pose for joint configuration \(q\) |
| \(J(q)\) | Jacobian: how the end-effector moves for a small change of each joint |
| C-space | Configuration space: the set of all joint configurations |
| Workspace, task space | Points the end effector can reach, and the space the task is defined in |
| \(f(s)\), \(s \in [0, 1]\) | Time scaling of a move: \(s\) is the share of the move’s time used so far |
| \(e_k\), \(u_k\) | Tracking error and motor command at control step \(k\) |
| \(K_p\), \(K_i\), \(K_d\) | Proportional, integral and derivative gains of a PID controller |
| \(\pi_\theta(a_t \mid o_t)\) | Policy with parameters \(\theta\): a distribution over actions given the observation |
| \(\mathcal{M} = (\mathcal{S}, \mathcal{A}, P, R)\) | A Markov decision process: states, actions, transition probabilities, reward function |
| \(P(s_{t+1} \mid s_t, a_t)\) | Transition probability (dynamics); ADM writes \(T(s' \mid s, a)\) |
| \(r_t\), \(\gamma\) | Reward at step \(t\), and the discount factor in \([0, 1)\) |
| \(\tau\), \(p_\pi(\tau)\) | A trajectory \((s_0, a_0, s_1, \dots)\) and its probability under policy \(\pi\) |
| \(G\), \(J(\pi)\) | Return of one trajectory, and its expected value under \(\pi\) |
| \(\delta\), \(\nu\) | A random search direction, and the size of the nudge along it |
| Markov property | The next state depends only on the current state and action, not on earlier ones |
| Sparse, dense reward | Reward only on success, versus a reward that measures progress at every step |
Week 3 · Imitation Learning
| Symbol or term | Meaning |
|---|---|
| \(s_t\), \(o_t\), \(a_t\) | State, observation and action at time step \(t\) |
| \(\pi^*\) | The expert policy being imitated (ADM writes \(\pi_E\)) |
| \(\pi_\theta\) | The learned policy, a network with parameters \(\theta\) |
| \(\mathcal{D}\) | Demonstration data set of (state, expert action) pairs |
| \(c(s, a)\) | Cost of a step: 0 if \(a\) matches the expert’s action, 1 otherwise |
| \(\varepsilon\) | Probability that the learned policy makes a mistake on a state the expert visits |
| \(T\) | Task length (horizon) in steps |
| \(p_{\text{expert}}(s)\), \(p_{\pi_\theta}(s)\) | How often the expert, or the learned policy, visits state \(s\) |
| \(w_k, \mu_k, \Sigma_k\) | Weight, mean and covariance of component \(k\) of a mixture-of-Gaussians policy |
| \(a_{t,d}\) | Dimension \(d\) of the action, predicted as a bin in an autoregressive policy |
| \(z\) | Latent plan: a random input that selects one way of doing the task |
| \(s_c\), \(s_g\) | Current state and goal state in a goal-conditioned policy |
| \(q_\phi\), \(p(z \mid c)\) | Posterior (“plan recognition”) and prior (“plan proposal”) of a latent-variable policy |
| Behavioral cloning (BC) | Supervised learning of a policy from expert (state, action) pairs |
| DAgger | Dataset Aggregation: label the learner’s own states with expert actions, add them, retrain |
| Distribution shift | Test-time states differ from the states in the training data |
| Causal confusion | A policy relies on a correlate of the expert’s action instead of its cause |
Week 4 · Reinforcement Learning I
| Symbol or term | Meaning |
|---|---|
| \(V^\pi(s)\) | Value of state \(s\): expected discounted return when starting there and following \(\pi\) (ADM writes \(U^\pi\)) |
| \(V^*(s)\), \(\pi^*\) | Optimal value function, and a policy that achieves it |
| \(Q^\pi(s, a)\), \(Q^*(s, a)\) | Action value: expected return after taking \(a\) in \(s\), then following \(\pi\) (or acting optimally) |
| \(\mathcal{T}\) | Bellman (optimality) operator: one sweep of value iteration |
| TD target | \(r + \gamma \max_{a'} Q(s', a')\) (Q-learning) or \(r + \gamma Q(s', a')\) (SARSA): a one-sample estimate of \(Q(s, a)\) |
| \(\alpha\) | Learning rate of a tabular update |
| \(\varepsilon\)-greedy | Act randomly with probability \(\varepsilon\), otherwise take the best action |
| On-policy, off-policy | Learning the value of the policy that collects the data, versus of another (usually the greedy) policy |
| \(\theta\), \(\theta^-\) | Weights of the Q-network and of its slowly updated target copy |
| \(\mathcal{D}\) | Replay buffer of stored transitions \((s, a, r, s')\) |
| \(\mu_\phi(s)\) | DDPG’s actor: a network that outputs the action directly |
| CEM | Cross-entropy method: sample, keep the best, refit a Gaussian, repeat |
Week 5 · Reinforcement Learning II
| Symbol or term | Meaning |
|---|---|
| \(\pi_\theta(a \mid s)\) | A policy with parameters \(\theta\), here a distribution over actions |
| \(\tau\), \(p_\theta(\tau)\) | A trajectory \((s_0, a_0, s_1, \dots)\) and its probability under \(\pi_\theta\) and the dynamics |
| \(R(\tau)\), \(J(\theta)\) | A trajectory’s return, and the expected return of the policy (ADM writes \(U(\theta)\)) |
| \(\nabla_\theta \log \pi_\theta(a \mid s)\) | Score function: the direction that makes action \(a\) more likely |
| Reward-to-go | \(\sum_{t' \ge t} r_{t'}\): the rewards from step \(t\) to the end |
| \(b\) | Baseline subtracted from the return; may depend on the state, never on the action |
| \(A^\pi(s, a)\) | Advantage \(Q^\pi(s, a) - V^\pi(s)\): how much better \(a\) is than the policy’s average |
| \(\theta\), \(\theta'\) | The policy that collected the data, and the new policy being optimized |
| \(\rho\) | Importance ratio \(\pi_{\theta'}(a \mid s) / \pi_\theta(a \mid s)\) |
| \(\mathcal{H}(\pi)\) | Entropy of the policy’s action distribution |
| \(D_{\text{KL}}\) | KL divergence between two distributions |
| \(\epsilon\) | PPO’s clip range: ratios are clipped to \([1 - \epsilon, 1 + \epsilon]\) |
| \(\beta\), \(\alpha\) | Entropy temperature in the lecture’s entropy bonus (\(\beta\)) and in SAC (\(\alpha\)) |
| \(\delta\) | TRPO’s trust-region size, a bound on the average KL divergence |
| \(Q_\phi\), \(Q_{\bar\phi}\) | SAC’s critic and its slowly updated target copy |
| \(\mu_\theta(s)\), \(\sigma_\theta(s)\), \(\varepsilon\) | Mean, standard deviation and fixed noise of a reparameterized action \(a = \mu + \sigma\varepsilon\) |
Week 6 · Generative Models
| Symbol or term | Meaning |
|---|---|
| \(x_0\) | A clean data point: an action, an action sequence, or an image |
| \(x_i\), \(i = 1, \dots, T\) | The same point after \(i\) noising steps; \(T\) steps in all |
| \(\beta_i\), \(\alpha_i = 1 - \beta_i\) | Noise schedule: how much noise step \(i\) adds, and how much signal it keeps |
| \(\bar\alpha_i\) | Signal kept after \(i\) steps: the product of \(\alpha_1, \dots, \alpha_i\) |
| \(\varepsilon\), \(\varepsilon_\theta(x_i, i)\) | Standard Gaussian noise, and the network’s prediction of it |
| \(z\), \(p(z)\) | A latent code and its prior, usually \(\mathcal{N}(0, I)\) |
| \(q_\phi(z \mid x)\), \(p_\theta(x \mid z)\) | Encoder and decoder of a VAE |
| \(e_k\), \(\mathrm{sg}[\cdot]\) | Codebook vector \(k\) of a VQ-VAE, and the stop-gradient operator |
| \(w\), \(c\), \(\varnothing\) | Guidance scale, condition, and the “no condition” token |
| \(t \in [0, 1]\), \(v_\theta(x_t, t)\) | Flow-matching time (noise at 0, data at 1) and the learned velocity |
| DDPM, DDIM | Diffusion sampling with fresh noise at every step, and deterministic sampling with fewer steps |
Week 7 · Sequence Modeling and Transformers
| Symbol or term | Meaning |
|---|---|
| \(x_{1:i-1}\), \(x_{<t}\) | The prefix of a sequence: every element before position \(i\) (or \(t\)) |
| \(p_\theta(x_i \mid x_{1:i-1})\) | An autoregressive model’s conditional: the next element given all earlier ones |
| \(h_t\) | Hidden state of a recurrent network after step \(t\) |
| \(Q, K, V\); \(q_i, k_j, v_j\) | Queries, keys and values (matrices, and one token’s vectors) |
| \(W_Q, W_K, W_V\) | Learned projections from token embeddings to queries, keys and values |
| \(d\) | Width of the query and key vectors; scores are divided by \(\sqrt{d}\) |
| \(s_{ij}\), \(\alpha_{ij}\) | Attention score of token \(i\) for token \(j\), and its softmax weight |
| \(\mathrm{PE}(i)\), \(b(i - j)\) | Absolute positional encoding of position \(i\); relative bias for distance \(i - j\) |
| \(\lambda\) | Slope of the goal toy’s relative bias, \(b(i - j) = -\lambda (i - j)\) |
| Teacher forcing | Training on the true previous tokens instead of the model’s own outputs |
| BPE | Byte-pair encoding: merge the most frequent adjacent pair, \(k\) times |
| \(N\), \(D\), \(C\) | Parameters, training tokens and training compute (\(C \approx 6ND\)) |
| \(s_{ij}\), \(\tau\) (CLIP) | Image-caption cosine similarity and the softmax temperature |
| \(\alpha\) (Flamingo) | A layer’s gate: cross-attention output is scaled by \(\tanh(\alpha)\), with \(\alpha = 0\) at the start |
| \(q_{01}\), \(q_{99}\) | 1st and 99th percentiles of an action dimension, used to normalize before binning |
| \(k\), \(a_{t:t+k}\) | Chunk length, and the chunk of actions a policy predicts at once |
| \(A_t[i]\), \(w_i\), \(m\) | Predictions stored for step \(t\) (\(i = 0\) the oldest), their weights \(e^{-mi}\), and the decay rate |
| \(c_j\), \(q_j\) | DCT coefficient for frequency \(j\), and its rounded, scaled value |
Week 8 · World Models
| Symbol or term | Meaning |
|---|---|
| \(p(s_{t+1} \mid s_t, a_t, h_t)\) | A world model: the next state given state, action and memory |
| \(\pi(a_t \mid s_t, g)\) | A goal-conditioned policy, for contrast |
| \(\hat s_{t+k}\) | A state predicted (imagined) \(k\) steps ahead by the model |
| \(H\) | Planning or imagination horizon, in steps |
| \(c(\hat s)\) | Cost of an imagined state in model predictive control |
| \(\varepsilon\), \(L\), \(e_k\) | One-step model error, how much the dynamics stretch an error, and the error after \(k\) imagined steps |
| \(o_t\), \(z_t\), \(h_t\) | Observation, its latent code, and the recurrent memory |
| \(q(z_t \mid h_t, o_t)\), \(p(z_t \mid h_t)\) | RSSM posterior (sees the observation) and prior (memory only) |
| \(\pi_i, \mu_i, \sigma_i\) | Weight, mean and spread of a mixture component in World Models’ memory network |
| \(v_\psi\) | Dreamer’s critic, a value estimate of imagined states |
| \(c_s\), \(c_t\) | A video tokenizer’s compression factor in each spatial direction and in time |
| \(\ell\), \(g_\psi\) | Language instruction; inverse dynamics model that maps observations to actions |
| \(\mathrm{sg}[\cdot]\), \(\bar\theta\) | Stop-gradient, and the target encoder’s weights (an EMA or a frozen copy) |
| MPC, CEM | Model predictive control; the cross-entropy method used to search plans |
| WAM, VAM, IDM | World action model; video action model; inverse dynamics model |
| JEPA | Joint-embedding predictive architecture: predicts embeddings, not pixels |
Week 9 · Generalist Robot Policies
| Symbol or term | Meaning |
|---|---|
| \(g\) | A goal: a goal image or a language instruction \(\ell\) |
| \(e\), \(d_e\), \(D\) | An embodiment, its number of action dimensions, and the largest \(d_e\) (the padded size) |
| \(m_{ij}\) | Loss mask: 1 for a real action entry, 0 for padding or another robot’s head |
| \(f_k\) | Multicontext imitation: the encoder of task descriptions of kind \(k\) into a shared goal space |
| \(m(b, l)\) | Gato’s loss mask: 1 if token \(l\) of sequence \(b\) is text or an action |
| \(\mu\), \(M\) | Mu-law parameters (Gato: \(\mu = 100\), \(M = 256\)) |
| \(L_i\), \(B\) | Token length of sample \(i\); samples in a batch |
| \(c_i\) (cu_seqlens) | Where sample \(i\) starts in a packed sequence: the sum of the lengths before it |
| \(\mathrm{sg}[\cdot]\) | Stop-gradient: the action expert reads the backbone but sends it no gradient |
| \(A\), \(I_t\), \(\epsilon_\ell\) | Advantage, the improvement indicator \(\mathbb{1}[A > \epsilon_\ell]\), and its task threshold (π*0.6) |
| \(R_i\), \(R_{S,i}\) | Real and simulated success rate of policy \(i\) |
| MMRV | Mean maximum rank violation: how badly a simulator misorders policies, weighted by their real gap |
| SysID | System identification: fitting a simulator’s parameters to measurements of the real robot |
| VLA | Vision-language-action model: a VLM fine-tuned to output actions |
| OXE | Open X-Embodiment, the pooled data set of over a million real-robot episodes |
Week 10 · Embodied Reasoning and Test-time Scaling
| Symbol or term | Meaning |
|---|---|
| \(c\) | The reasoning chain written before the action (plan, subtask, move, objects, gripper) |
| \(p_{\text{drop}}\), \(m\) | Reasoning-dropout probability, and the mask that hides the reasoning (\(m = 0\)) |
| ECoT, ECoT-Lite | Embodied chain-of-thought; its recipes that write no reasoning at test time |
| \(N\) | Number of samples drawn at test time (best-of-N) |
| \(p\) | Chance that one sample (or one answer) is right |
| \(V(o, a)\) | A verifier’s score for candidate \(a\) |
| \(\sigma_V\), \(w\) | The toy verifier’s error on the gap position; the gap’s width |
| \(E_A\), \(\Delta\) | An Elo rating; a rating gap |
| \(L\), \(T\) | Transformer layers; reasoning tokens written before the answer |
| \(q\), \(o_i\), \(G\) | A question, the \(i\)-th sampled answer, and the group size (GRPO) |
| \(r_i\), \(\hat A_{i,t}\) | Answer \(i\)’s reward, and its group-relative advantage, shared by all its tokens \(t\) |
| \(\rho_{i,t}\) | Probability ratio of token \(t\) of answer \(i\), new policy over old |
| \(\beta\), \(\pi_{\text{ref}}\) | KL weight and the frozen reference policy |
| GRPO | Group relative policy optimization: PPO with a group-average baseline instead of a critic |
| Verifiable reward | A reward a rule can compute, such as checking the final answer |
Week 11 · Frontiers & Open Problems
| Symbol or term | Meaning |
|---|---|
| \(N(s, a)\) | How often the data take action \(a\) in state \(s\) |
| \(\mathcal{D}\), \(\tau\) | A data set of trajectories; one trajectory |
| \(g = s_{t'}\) | A goal relabelled in hindsight: a state the trajectory reached later |
| \(V(s)\) | Cost-to-go (steps to the goal) through transitions in the data |
| \(N_e\), \(M\), \(k\) | Expert episodes, autonomous episodes per round, rounds of the flywheel |
| \(\pi_m\), \(E\), \(u(o)\) | Ensemble member \(m\) of \(E\); the members’ disagreement on observation \(o\) |
| \(\tau\) (threshold) | Disagreement above which the robot asks for help |
| Coverage, selective success | Share of tasks the robot takes on; its success on those |
| \(x_i\), \(y_i\) | Normalized embeddings of image \(i\) and caption \(i\) |
| \(t\), \(b\) | Contrastive temperature; SigLIP’s bias |
| \(z_{ij}\) | SigLIP’s label: \(+1\) for a matching image-caption pair, \(-1\) otherwise |
| \(f\) | Fraction of the data with a property (a possible hard negative) |
| Stitching | Joining pieces of different trajectories into a new route |
| Data flywheel | Deploy, collect, retrain, deploy a better robot |
| Mid-training | Training after pre-training that adds core skills before fine-tuning |
Overloaded symbols
Symbols that mean different things in different weeks.
| Symbol | Meaning | Week |
|---|---|---|
| \(\varepsilon\) | Per-step mistake probability of a cloned policy | 3 |
| \(\varepsilon\) | Standard Gaussian noise added or predicted by a diffusion model | 6 |
| \(t\) | A time step of the robot’s trajectory | 1, 3 |
| \(t\) | Continuous flow-matching time from noise (\(t = 0\)) to data (\(t = 1\)) | 6 |
| \(\alpha_i\) | Signal kept by one diffusion step, \(1 - \beta_i\) | 6 |
| \(\alpha\) | Step size of a gradient or random-search update | 2 |
| \(\alpha\) | Learning rate of a tabular TD update | 4 |
| \(\varepsilon\) | Exploration probability of an \(\varepsilon\)-greedy policy | 4 |
| \(\mu\) | Mean of a Gaussian policy, or DDPG’s deterministic actor \(\mu_\phi(s)\) | 4 |
| \(T\) | A homogeneous transform (a \(3 \times 3\) or \(4 \times 4\) matrix) | 2 |
| \(T\) | Task length (horizon) in steps | 3 |
| \(T\) | Number of diffusion steps | 6 |
| \(f\) | Forward kinematics \(f(q)\), or a time scaling \(f(s)\); also a control frequency in Hz | 2 |
| \(J\) | The Jacobian \(J(q)\), or the expected return \(J(\pi)\) | 2 |
| \(J(\theta)\) | Expected return as a function of the policy’s parameters | 5 |
| \(\epsilon\) | PPO’s clip range | 5 |
| \(\varepsilon\) | Fixed Gaussian noise of a reparameterized action | 5 |
| \(\alpha\) | SAC’s entropy temperature | 5 |
| \(\beta\) | Entropy temperature of the lecture’s entropy bonus | 5 |
| \(\beta_i\) | Noise added by diffusion step \(i\) | 6 |
| \(\delta\) | TRPO’s trust-region size; on this page also the mean shift between two Gaussians, in standard deviations | 5 |
| \(\alpha_{ij}\) | Attention weight of token \(i\) on token \(j\) | 7 |
| \(\alpha\) | Flamingo’s cross-attention gate, used as \(\tanh(\alpha)\) | 7 |
| \(\tau\) | CLIP’s softmax temperature (elsewhere a trajectory) | 7 |
| \(k\) | Action-chunk length; also a number of BPE merges, and \(k_j\) a key vector | 7 |
| \(N\), \(D\) | Parameters and training tokens in scaling laws; also \(N\) samples per chunk in the DCT | 7 |
| \(q_j\), \(q_i\) | A rounded DCT coefficient, or a query vector | 7 |
| \(\pi_i\) | A mixture weight in World Models’ memory network (not a policy) | 8 |
| \(H\) | Planning horizon in steps; also an image’s height in the video-token count | 8 |
| \(L\) | How much one step of the dynamics can stretch an error | 8 |
| \(\varepsilon\) | One-step error of a world model | 8 |
| \(q\) | RSSM posterior over the latent, \(q(z_t \mid h_t, o_t)\) | 8 |
| \(\mu\) | Mu-law compression parameter in Gato’s tokenizer (elsewhere a mean) | 9 |
| \(M\) | Mu-law range parameter; \(m\) is a loss mask | 9 |
| \(I_t\) | π*0.6’s improvement indicator, 1 when the advantage beats the threshold (not an identity matrix) | 9 |
| \(\epsilon_\ell\) | π*0.6’s advantage threshold for task \(\ell\) | 9 |
| \(o_i\) | The \(i\)-th sampled answer in a GRPO group (not an observation) | 10 |
| \(c\) | A reasoning chain (elsewhere a constant) | 10 |
| \(\beta\) | Weight of GRPO’s KL penalty to the reference policy | 10 |
| \(N\) | Samples drawn by best-of-N search | 10 |
| \(L\) | Number of transformer layers | 10 |
| \(T\) | Reasoning tokens written before the answer | 10 |
| \(\tau\) | A disagreement threshold (elsewhere a trajectory) | 11 |
| \(t\) | Contrastive temperature (elsewhere a time step) | 11 |
| \(M\) | Autonomous episodes per round of the flywheel | 11 |
| \(V(s)\) | Steps to the goal (a cost, minimized), not a reward-based value | 11 |