Notation
This site uses the book’s notation, so you can move between the two. Symbols are grouped by the Part where they first appear, in chapter order; each is listed once, where it is introduced, unless a later chapter gives it a new meaning. The book reuses some letters for different things in different chapters; the last section lists them.
For the key equations themselves, see the cheat sheets.
Introduction and Part I: probabilistic reasoning
| Symbol | Meaning | First used |
|---|---|---|
| \(r_t\) | Reward received at step \(t\) | Ch 1 |
| \(a^*\) | The action with the highest expected utility | Ch 1 |
| \(P(x)\) | Probability mass function (discrete): the probability that \(X = x\) | Ch 2 |
| \(p(x)\) | Probability density function (continuous); probabilities are areas under it | Ch 2 |
| \(\operatorname{cdf}(x)\) | Cumulative distribution function, \(P(X \le x)\) | Ch 2 |
| \(\mathcal{N}(x \mid \mu, \sigma^2)\) | Gaussian with mean \(\mu\) and variance \(\sigma^2\); \(\mathcal{N}(\mathbf{x} \mid \boldsymbol\mu, \boldsymbol\Sigma)\) is the multivariate version | Ch 2 |
| \(\mathcal{U}(x \mid a, b)\) | Uniform distribution on \([a, b]\) | Ch 2 |
| \(\rho_i\) | Weight of component \(i\) in a mixture | Ch 2 |
| \(x^0\), \(x^1\) | The two values of a binary variable: no, yes | Ch 2 |
| \(P(x \mid y)\) | Conditional probability of \(x\) given \(y\) | Ch 2 |
| \(\theta_1\), \(\theta_2\) | Threshold and softness of a sigmoid conditional | Ch 2 |
| \(x_{1:n}\) | The values \(x_1, \ldots, x_n\) | Ch 2 |
| \(\mathrm{pa}(x_i)\) | Values of the parents of \(X_i\) in a Bayesian network | Ch 2 |
| \((X \perp Y \mid Z)\) | \(X\) and \(Y\) are conditionally independent given \(Z\) | Ch 2 |
| \(q\), \(e\), \(h\) | Query, evidence, and hidden variables in an inference problem | Ch 3 |
| \(\phi\) | A factor: a table over some variables (a CPT, or any partial product) | Ch 3 |
| \(P(c \mid o_{1:n})\) | Naive Bayes posterior of class \(c\) given observations \(o_1, \ldots, o_n\) | Ch 3 |
| \(w\) | Likelihood weight of a sample | Ch 3 |
| \(x_{-i}\) | All variables except \(x_i\) | Ch 3 |
| \(\boldsymbol{\mu}_{a \mid b}\), \(\boldsymbol{\Sigma}_{a \mid b}\) | Mean and covariance of Gaussian block \(\mathbf{a}\) given block \(\mathbf{b}\) | Ch 3 |
| \(\theta\) | Model parameters, such as CPT entries | Ch 4 |
| \(\ell(\theta)\) | Log-likelihood of the data under parameters \(\theta\) | Ch 4 |
| \(m_{ijk}\) | Count of records with variable \(i\) at value \(k\) and its parents at combination \(j\) | Ch 4 |
| \(\hat\theta\) | An estimate of \(\theta\) | Ch 4 |
| \(\mathrm{Beta}(\theta \mid \alpha, \beta)\) | Beta distribution over a probability \(\theta\); \(\alpha\), \(\beta\) act as pseudocounts | Ch 4 |
| \(\mathrm{Dir}(\theta_{1:n} \mid \alpha_{1:n})\) | Dirichlet distribution over \(n\) probabilities that sum to 1 | Ch 4 |
| \(\phi\), \(h\) | Kernel and bandwidth in kernel density estimation | Ch 4 |
| \(G\), \(D\) | A graph (network structure) and a dataset | Ch 5 |
| \(\ln P(G \mid D)\) | Bayesian score of graph \(G\) on data \(D\) | Ch 5 |
| \(\Gamma(x)\) | Gamma function; \(\Gamma(n) = (n-1)!\) for whole numbers | Ch 5 |
| \(\alpha_{ijk}\), \(\alpha_{ij0}\), \(m_{ij0}\) | Pseudocount for a cell, and the row totals of pseudocounts and counts | Ch 5 |
| \(q_i\), \(r_i\) | Number of parent combinations and number of values of variable \(i\) | Ch 5 |
| \(\kappa\) | Equivalent sample size of a BDeu prior | Ch 5 |
| \(\mathrm{pa}_G(x_i)\) | Parents of \(x_i\) in graph \(G\) | Ch 5 |
| \(A \succ B\), \(A \sim B\), \(A \succeq B\) | \(A\) preferred to \(B\); indifferent; at least as good | Ch 6 |
| \([S_1 : p_1; \ldots; S_n : p_n]\) | Lottery: outcome \(S_i\) with probability \(p_i\) | Ch 6 |
| \(U(S)\) | Utility of outcome \(S\) | Ch 6 |
| \(S_\top\), \(S_\bot\) | Best and worst outcomes, used to scale utility to \([0, 1]\) | Ch 6 |
| \(x_{\text{ce}}\) | Certainty equivalent: the sure amount worth the same as a gamble | Ch 6 |
| \(\lambda\) | Risk-aversion parameter of exponential utility \(1 - e^{-\lambda x}\) | Ch 6 |
| \(EU(a \mid o)\), \(EU^*(o)\) | Expected utility of action \(a\) given observation \(o\); its maximum over actions | Ch 6 |
| \(\mathrm{VOI}(O' \mid o)\) | Value of information of observing \(O'\), given \(o\) | Ch 6 |
Part II: sequential problems
| Symbol | Meaning | First used |
|---|---|---|
| \(\mathcal{S}\) | State space | Ch 7 |
| \(\mathcal{A}\) | Action space | Ch 7 |
| \(T(s' \mid s, a)\) | Transition model: probability of reaching \(s'\) from \(s\) after action \(a\) | Ch 7 |
| \(R(s, a)\) | Expected reward for taking action \(a\) in state \(s\) | Ch 7 |
| \(\gamma\) | Discount factor, \(0 \le \gamma \le 1\) | Ch 7 |
| \(\pi\) | Policy: maps states to actions | Ch 7 |
| \(U^\pi(s)\) | Value (utility) of state \(s\) under policy \(\pi\) | Ch 7 |
| \(U^*(s)\) | Optimal value of state \(s\) | Ch 7 |
| \(U_k\) | Value estimate after \(k\) sweeps of value iteration | Ch 7 |
| \(Q(s, a)\) | Action value: reward for \(a\) in \(s\) plus the discounted value \(U\) of where you land | Ch 7 |
| \(A(s, a)\) | Advantage: \(Q(s, a) - U(s)\) | Ch 7 |
| \(\mathbf{T}^\pi\), \(\mathbf{R}^\pi\) | Transition matrix and reward vector with the policy’s action fixed in each state | Ch 7 |
| \(\lVert U \rVert_\infty\) | Largest absolute entry, \(\max_s \lvert U(s) \rvert\) | Ch 7 |
| \(\delta\), \(\epsilon\) | Bellman-residual threshold and target accuracy in value iteration | Ch 7 |
| \(U_\theta(s)\), \(Q_\theta(s, a)\) | Parametric (approximate) value and action value functions with parameters \(\theta\) | Ch 8 |
| \(S = s_{1:m}\) | Sample states where approximate value iteration does backups | Ch 8 |
| \(\beta(s)\) | Basis functions (features), or local weights, so that \(U_\theta(s) = \theta^\top \beta(s)\) | Ch 8 |
| \(d(s, s')\), \(k(s, s')\) | Distance between states, and a kernel (larger for closer states) | Ch 8 |
| \(\sigma\) | Width of a Gaussian kernel | Ch 8 |
| \(X\), \(X^+\) | Feature matrix (row \(i\) is \(\beta(s_i)^\top\)) and its pseudoinverse | Ch 8 |
| \(d\) | Search depth (planning horizon) of an online planner | Ch 9 |
| \(U_d(s)\) | Value of \(s\) from a depth-\(d\) search | Ch 9 |
| \(\underline{U}(s)\), \(\overline{Q}(s, a)\), \(\overline{U}(s)\) | Lower bound on values; upper bounds on action values and values (admissible heuristic) | Ch 9 |
| \(m\) | Number of samples (sparse sampling) or simulations (MCTS, heuristic search) | Ch 9 |
| \(N(s, a)\), \(N(s)\) | Visit counts in Monte Carlo tree search | Ch 9 |
| \(c\) | Exploration constant in UCB1 | Ch 9 |
| \(\delta\) | Residual threshold for a state to count as solved | Ch 9 |
| \(a_{1:d}\) | An open-loop sequence of actions | Ch 9 |
| \(\pi_\theta\), \(U(\theta)\) | A policy with parameters \(\theta\), and its expected return | Ch 10 |
| \(\tau\), \(R(\tau)\), \(p_\pi(\tau)\) | A trajectory, its discounted return, and its probability under \(\pi\) | Ch 10 |
| \(b(s)\) | Initial state distribution (for policy search) | Ch 10 |
| \(\alpha\), \(c\), \(\epsilon\) | Step size, its reduction factor, and the stopping step size (Hooke–Jeeves) | Ch 10 |
| \(m_{\text{elite}}\) | Number of elite samples kept each iteration | Ch 10 |
| \(p(\theta \mid \psi)\) | Search distribution over policy parameters, with parameters \(\psi\) | Ch 10 |
| \(w^{(i)}\) | Rank-shaped weight of the \(i\)-th best sample | Ch 10 |
| \(\nabla U(\theta)\) | Policy gradient: how the expected return changes with each parameter | Ch 11 |
| \(\delta\), \(e_i\) | Finite-difference step, and the unit vector along parameter \(i\) | Ch 11 |
| \(\Delta\Theta\), \(\Delta U\) | Matrix of parameter perturbations (one per row) and the resulting changes in \(U\) | Ch 11 |
| \(\pi_\theta(a \mid s)\) | Probability (or density) that the policy takes action \(a\) in state \(s\) | Ch 11 |
| \(\nabla_\theta \log \pi_\theta(a \mid s)\) | Score of an action: the direction that makes it more likely | Ch 11 |
| \(r_{\text{to-go}}^{(k)}\) | Reward-to-go: discounted reward from step \(k\) on | Ch 11 |
| \(r_{\text{base}}(s)\) | Baseline subtracted from the reward-to-go | Ch 11 |
| \(F_\theta\) | Fisher information matrix of the policy’s trajectory distribution | Ch 12 |
| \(D_{\text{KL}}(p \,\Vert\, q)\) | Kullback–Leibler divergence from \(p\) to \(q\) | Ch 12 |
| \(\epsilon\) | Trust-region size (KL or step budget); PPO’s clipping range | Ch 12 |
| \(f(\theta, \theta')\), \(g(\theta, \theta')\) | TRPO’s surrogate objective and surrogate KL constraint | Ch 12 |
| \(b_{\gamma,\theta}(s)\) | Discounted state-visitation distribution of \(\pi_\theta\) | Ch 12 |
| \(w = \pi_{\theta'}(a \mid s) / \pi_\theta(a \mid s)\) | Probability ratio (importance weight) | Ch 12 |
| \(U_\phi(s)\), \(Q_\phi(s, a)\) | The critic: a value function with parameters \(\phi\) | Ch 13 |
| \(\delta_k\) | Temporal difference residual \(r_k + \gamma U(s_{k+1}) - U(s_k)\) | Ch 13 |
| \(\hat A^{(k)}\), \(\hat A^{\text{GAE}}\) | \(k\)-step and generalized advantage estimates | Ch 13 |
| \(\lambda\) | GAE weight between TD residuals (\(\lambda = 0\)) and full returns (\(\lambda = 1\)) | Ch 13 |
| \(\nabla_\theta \pi_\theta(s)\) | Jacobian of a deterministic policy (one column per action component) | Ch 13 |
| \(\pi_{\text{MCTS}}\), \(U_{\text{MCTS}}\), \(\eta\) | Tree search’s visit-count policy (sharpened by \(\eta\)) and value | Ch 13 |
| \(f(\pi)\) | A performance metric of policy \(\pi\): the expected trajectory metric | Ch 14 |
| \(f_{\text{traj}}\), \(f_{\text{step}}\), \(f_{\text{state}}\) | Metric of a trajectory, of one step, and its expected total from a state | Ch 14 |
| \(b(s_1)\) | Start (initial state) distribution; in Ch 19 on, \(b\) is a belief | Ch 14 |
| \(\text{SE} = \hat\sigma/\sqrt n\) | Standard error of a sample mean; divided by \(\hat\mu\), the relative standard error | Ch 14 |
| \(P'(\tau)\), \(P^*(\tau)\) | Proposal distribution for importance sampling; the optimal (zero-variance) proposal | Ch 14 |
| \(w = P(\tau)/P'(\tau)\) | Importance weight of a trajectory sampled from the proposal | Ch 14 |
| \(T_i\), \(R_i\) | The \(i\)-th of several candidate models in robust dynamic programming | Ch 14 |
| \(R'\), \(\lambda\) | Adversarial reward, and its weight on the log-likelihood of the adversary’s choices | Ch 14 |
Part III: model uncertainty
| Symbol | Meaning | First used |
|---|---|---|
| \(\theta_a\), \(\theta^*\) | Payoff probability of bandit arm \(a\); the best arm’s | Ch 15 |
| \(w_a\), \(\ell_a\) | Wins and losses observed on arm \(a\) | Ch 15 |
| \(\rho_a\) | Posterior probability that the next pull of arm \(a\) wins, \((w_a + 1)/(w_a + \ell_a + 2)\) | Ch 15 |
| \(\varepsilon\), \(\lambda\), \(\alpha\), \(c\) | Exploration parameters: ε-greedy rate, softmax precision, quantile level, UCB1 constant | Ch 15 |
| \(N(a)\), \(N\) | Pulls of arm \(a\) (here Beta pseudocounts), and their total | Ch 15 |
| \(U^*_r\), \(Q^*_r\) | Bayes-optimal bandit values with \(r\) pulls remaining | Ch 15 |
| \(N(s, a, s')\), \(N(s, a)\) | Transition counts, and how often \(a\) was tried in \(s\) | Ch 16 |
| \(\rho(s, a)\) | Sum of rewards received for \(a\) in \(s\) | Ch 16 |
| \(\hat T\), \(\hat R\) | Estimated (maximum likelihood) transition and reward functions | Ch 16 |
| \(m\), \(r_{\max}\) | R-MAX’s visit threshold and the largest possible reward | Ch 16 |
| \(\mathrm{Dir}(\theta \mid \boldsymbol\alpha)\) | Dirichlet distribution over probability vectors, with pseudocounts \(\boldsymbol\alpha\) | Ch 16 |
| \(b\), \(\tau(s, b, a, s')\) | Belief over models, and its Bayes update in a Bayes-adaptive MDP | Ch 16 |
| \(\alpha(m)\), \(\alpha\) | Learning rate: how far an estimate moves toward each new sample | Ch 17 |
| \(\hat x_m\) | Incremental estimate of a mean after \(m\) samples | Ch 17 |
| \(\delta\) | Temporal difference error, e.g. \(r + \gamma Q(s', a') - Q(s, a)\) | Ch 17 |
| \(N(s, a)\) (traces) | Eligibility trace: a decaying count of recent visits (not the Ch 16 transition count) | Ch 17 |
| \(\lambda\) (traces) | Trace decay parameter in Sarsa(λ) | Ch 17 |
| \(F(s, a, s')\), \(\beta(s)\) | Shaping reward, and the potential function that makes it safe | Ch 17 |
| \(Q_\theta(s, a)\) | Parameterized action value function | Ch 17 |
| \(m_{\text{grad}}\) | Batch size for experience replay | Ch 17 |
| \(\mathcal{D}\) | Expert demonstrations (state-action pairs or trajectories) | Ch 18 |
| \(\pi_E\) | The expert’s policy | Ch 18 |
| \(\beta(s, a)\), \(\phi\) | Reward features and weights: \(R_\phi(s, a) = \phi^\top\beta(s, a)\) | Ch 18 |
| \(\mu_\pi\), \(\mu_E\) | Feature expectations of a policy, and of the expert | Ch 18 |
| \(t\) | Margin in maximum margin IRL | Ch 18 |
| \(\lambda_i\) | Mixture weights over the policies found by maximum margin IRL | Ch 18 |
| \(\beta\) (SMILe) | Mixing rate in SMILe (not the features) | Ch 18 |
| \(P_\phi(\tau)\), \(Z(\phi)\) | Maximum entropy trajectory distribution and its normalizer | Ch 18 |
| \(C_\phi(s, a)\) | GAIL discriminator: probability a pair came from the learner | Ch 18 |
Part IV: state uncertainty
| Symbol | Meaning | First used |
|---|---|---|
| \(o\) | An observation | Ch 19 |
| \(O(o \mid a, s')\) | Observation model: probability of observing \(o\) after taking \(a\) and landing in \(s'\) | Ch 19 |
| \(b(s)\) | Belief: probability that the current state is \(s\), given the history | Ch 19 |
| \(\mathcal{N}(x \mid \mu, \Sigma)\) | Gaussian density at \(x\) with mean \(\mu\) and covariance \(\Sigma\) | Ch 19 |
| \(\mathbf{T}_s\), \(\mathbf{T}_a\), \(\boldsymbol{\Sigma}_s\) | Linear-Gaussian dynamics: state matrix, action matrix, process-noise covariance | Ch 19 |
| \(\mathbf{O}_s\), \(\boldsymbol{\Sigma}_o\) | Linear-Gaussian sensor: observation matrix, sensor-noise covariance | Ch 19 |
| \(\boldsymbol{\mu}_b\), \(\boldsymbol{\Sigma}_b\) | Mean and covariance of a Gaussian belief | Ch 19 |
| \(\boldsymbol{\mu}_p\), \(\boldsymbol{\Sigma}_p\) | Mean and covariance after the predict step | Ch 19 |
| \(\mathbf{K}\) | Kalman gain | Ch 19 |
| \(s^{(i)}\), \(w^{(i)}\) | The \(i\)-th particle and its weight | Ch 19 |
| \(\text{Update}(b, a, o)\) | The belief after acting and observing (the Ch 19 filter) | Ch 20 |
| \(R(b, a)\), \(T(b' \mid b, a)\) | Reward and transition of the belief-state MDP | Ch 20 |
| \(P(o \mid b, a)\) | Probability of observing \(o\) after taking \(a\) from belief \(b\) | Ch 20 |
| \(\pi()\), \(\pi(o)\) | A conditional plan’s root action, and its subplan after observation \(o\) | Ch 20 |
| \(\alpha_\pi\), \(\Gamma\) | Alpha vector of a plan (utility per state), and a set of them | Ch 20 |
| \(U^\Gamma(b)\) | \(\max_{\alpha \in \Gamma} \alpha^\top b\), the value represented by \(\Gamma\) | Ch 20 |
| \(\delta\) (pruning) | Largest lead of an alpha vector over a set, over all beliefs | Ch 20 |
| \(\mu_b\), \(K\) | Belief mean and linear feedback gain in LQG control | Ch 20 |
| \(U_{\text{hi}}\), \(U_{\text{lo}}\) | Upper and lower bounds on the optimal value over beliefs | Ch 21 |
| \(r_{\text{baws}}\) | Best-action worst-state lower bound | Ch 21 |
| \(B\) | A finite set of beliefs used by point-based methods | Ch 21 |
| \(\alpha_{a,o}\) | The best existing alpha vector after action \(a\) and observation \(o\) | Ch 21 |
| \(V\), \(u_E\) | Stored belief-utility pairs (sawtooth), and the corner beliefs’ utilities | Ch 21 |
| \(\rho(b, b')\) | Minimum ratio \(\min_i b_i/b'_i\) in the sawtooth bound | Ch 21 |
| \(m\), \(\lambda_i\) | Grid granularity and interpolation weights of a triangulation | Ch 21 |
| \(\delta\) (gap) | Gap below which heuristic search stops refining a belief | Ch 21 |
| \(Q_d(b, a)\), \(U_d(b)\) | Forward-search action value and value with \(d\) steps left | Ch 22 |
| \(m\) (online) | Samples per action (sparse sampling), simulations (MCTS) or scenarios (determinized search) | Ch 22 |
| \(h\) | A history: the actions and observations since the current belief | Ch 22 |
| \(N(h, a)\), \(Q(h, a)\) | Visit count and value estimate of action \(a\) after history \(h\) (MCTS) | Ch 22 |
| \(\Phi\), \(\phi = (s, i, j)\) | Determinizing matrix of uniform numbers; a particle’s state, scenario and depth | Ch 22 |
| \(X\), \(x\) | A controller’s nodes, and one node (its memory state, not the environment’s state) | Ch 23 |
| \(\psi(a \mid x)\) | A controller’s action distribution at node \(x\) | Ch 23 |
| \(\eta(x' \mid x, a, o)\) | A controller’s successor distribution after action \(a\) and observation \(o\) | Ch 23 |
| \(U(x, s)\), \(\alpha_x\) | Value of node \(x\) in state \(s\); node \(x\)’s alpha vector | Ch 23 |
| \(T_\theta\), \(r_\theta\), \(u_\theta\) | Transition matrix, rewards and values on (node, state) pairs for controller parameters \(\theta = (\psi, \eta)\) | Ch 23 |
| \(\beta\), \(v\) | Start distribution on (node, state) pairs; expected discounted time spent in each pair | Ch 23 |
| \(\tau\) | Shift in the projection onto the probability simplex | Ch 23 |
Part V: multiagent systems
| Symbol | Meaning | First used |
|---|---|---|
| \(\mathcal{I}\), \(i\), \(k\) | The agents, one agent, and their number | Ch 24 |
| \(\mathcal{A}^i\), \(\boldsymbol{a} = (a^1, \dots, a^k)\) | Agent \(i\)’s actions; a joint action | Ch 24 |
| \(-i\) | Every agent except \(i\), as in \(\boldsymbol{a}^{-i}\) or \(\boldsymbol{\pi}^{-i}\) | Ch 24 |
| \(R^i(\boldsymbol{a})\) | Agent \(i\)’s reward for the joint action \(\boldsymbol{a}\) | Ch 24 |
| \(\pi^i(a^i)\), \(\boldsymbol{\pi}\) | Agent \(i\)’s policy (a pure or mixed strategy); the joint policy | Ch 24 |
| \(U^i(\boldsymbol{\pi})\) | Agent \(i\)’s expected reward under the joint policy | Ch 24 |
| \(\pi(\boldsymbol{a})\) (correlated) | A single distribution over joint actions | Ch 24 |
| \(\lambda\), \(k\) (hierarchical softmax) | Precision of the softmax response; level of reasoning | Ch 24 |
| \(N^i(j, a^j)\) | Agent \(i\)’s count of how often agent \(j\) played \(a^j\) (fictitious play) | Ch 24 |
| \(\pi^i(a^i \mid s)\) | Agent \(i\)’s state-dependent policy in a Markov game | Ch 25 |
| \(R^i(s, \boldsymbol{a})\), \(T(s' \mid s, \boldsymbol{a})\) | Agent \(i\)’s reward and the shared transition model, given the joint action | Ch 25 |
| \(U^{\boldsymbol{\pi}, i}(s)\) | Agent \(i\)’s value of state \(s\) under the joint policy | Ch 25 |
| \(Q^i(s, \boldsymbol{a})\) | Agent \(i\)’s value of a joint action in state \(s\) | Ch 25 |
| \(N(j, a^j, s)\) | Count of agent \(j\) playing \(a^j\) in state \(s\) (fictitious play) | Ch 25 |
| \(\boldsymbol{o} = (o^1, \dots, o^k)\), \(O(\boldsymbol{o} \mid \boldsymbol{a}, s')\) | A joint observation, and its probability after joint action \(\boldsymbol{a}\) | Ch 26 |
| \(\boldsymbol{\pi}()\), \(\boldsymbol{\pi}(\boldsymbol{o})\) | Joint conditional plan’s root actions; each agent’s subplan after its own observation | Ch 26 |
| \(\Pi^i_h\) | Agent \(i\)’s set of \(h\)-step conditional plans | Ch 26 |
| \(b(\boldsymbol{\pi}^{-i}, s)\) | Distribution over the other agents’ plans and the state (pruning LP) | Ch 26 |
| \(\boldsymbol{x} = (x^1, \dots, x^k)\) | Joint controller node | Ch 26 |
| \(R(s, \boldsymbol{a})\) (shared) | The team’s single reward in a Dec-POMDP | Ch 27 |
| \(U^*_{\text{MPOMDP}}(b)\) | Optimal team value if observations were shared (an upper bound) | Ch 27 |
Appendices
| Symbol | Meaning | First used |
|---|---|---|
| \(\lVert \mathbf{x} \rVert_p\) | The \(p\)-norm of a vector; \(\lVert \mathbf{x} \rVert_\infty\) is its largest absolute entry | App A |
| \(\mathbf{A} \succ 0\) | \(\mathbf{A}\) is positive definite: \(\mathbf{x}^\top \mathbf{A} \mathbf{x} > 0\) for every \(\mathbf{x} \ne \mathbf{0}\) | App A |
| \(H(P)\), \(H(P, Q)\) | Entropy of \(P\); cross entropy of \(Q\) relative to \(P\) | App A |
| \(\nabla f(\mathbf{x})\) | Gradient: the vector of partial derivatives, pointing uphill | App A |
| \(B\) | The Bellman backup as an operator on value functions | App A |
| \(O(g(n))\), \(\Omega(g(n))\), \(\Theta(g(n))\) | Grows no faster than, no slower than, and as fast as \(g\), up to constant factors | App C |
| \(\mathbf{W}\), \(\mathbf{b}\), \(\phi\) | A network layer’s weights, biases and activation function | App D |
| \(\operatorname{softmax}(\mathbf{z})\) | Turns scores into probabilities: \(e^{z_i}/\sum_j e^{z_j}\) | App D |
| \(\beta\) (regularization) | Weight of the penalty on squared network weights | App D |
| \(\mathcal{A}(s)\) | Valid actions in state \(s\) (none in an absorbing state) | App E |
| \(T(s, a)\) (deterministic) | The one next state after taking \(a\) in \(s\) | App E |
Symbols with several meanings
The book, and so this site, reuses a few letters. The chapter always makes the meaning clear, but when reading across chapters, check this list.
| Symbol | Meanings, by chapter |
|---|---|
| \(\alpha\) | Pseudocounts (Ch 4–5, 16); step size (Ch 10, App A); quantile level (Ch 15); learning rate (Ch 17); alpha vector (Ch 20–23) |
| \(\beta\) | Beta parameter (Ch 4); basis functions (Ch 8); shaping potential (Ch 17); reward features and SMILe rate (Ch 18); start distribution (Ch 23); regularization weight (App D) |
| \(\delta\) | Residual thresholds (Ch 7, 9); finite-difference step (Ch 11); temporal difference residual (Ch 13, 17); pruning lead (Ch 20); belief gap (Ch 21) |
| \(\lambda\) | Risk aversion (Ch 6); GAE weight (Ch 13); adversary weight (Ch 14); softmax precision (Ch 15, 24); trace decay (Ch 17); interpolation and mixture weights (Ch 18, 21) |
| \(\phi\) | Factor (Ch 3); kernel (Ch 4); critic parameters (Ch 13); reward weights (Ch 18); particle (Ch 22); activation function (App D) |
| \(b\) | Initial state distribution (Ch 10, 14); belief over models (Ch 16); belief over states (Ch 19 on) |
| \(m\) | Counts (Ch 4–5); number of samples or simulations (Ch 9, 22); R-MAX threshold (Ch 16); grid granularity (Ch 21) |
| \(h\) | Hidden variables (Ch 3); kernel bandwidth (Ch 4); history (Ch 22); plan horizon (Ch 26) |
| \(N\) | Visit counts (Ch 9, 15, 16, 22); eligibility traces (Ch 17); opponent action counts (Ch 24, 25) |