MDL.X (deprecated) — How Deep Learning Changed How We Think about Intelligence

A useful dividing line is between structure that scales with compute and structure that substitutes for compute.

• Convolution or attention. These encode a symmetry or a routing mechanism, not domain facts, and they make learning cheaper without capping it, so they are arguably “meta-methods” in Sutton’s sense. Sutton’s own text is pulled both ways here. The evidence that architecture changes mostly shift scaling curves by a constant factor rather than change their slope, and that biases which help at small scale often stop helping at large scale (MDL4), suggests built-in structure that does get eaten, just slowly.
• Tokenizer. Hand-written, and visibly brittle on arithmetic and spelling. Byte-level models are what the bitter lesson predicts; whether they win at equal compute is still an empirical question.
• Data curation, RL rubrics, constitutions. These specify what we want, not how to compute it. The bitter lesson is a claim about capability, so these may be outside its scope: more compute cannot tell you which outcome is good. That is the alignment half of this workbook.

In the designed picture, the agent’s objective is a component someone writes down. The questions are whether we wrote the right one (outer alignment, in later terms) and whether it survives self-modification. Evolution versus human values was a known analogy (“adaptation-executers, not fitness-maximizers”), and Bostrom’s treacherous turn is close to deceptive alignment, but there is no second objective to compare the written one with.

The inner/outer split needs two optimizers: a training process, which optimizes parameters against a loss on a distribution, and a trained model, whose behaviour may be goal-directed in its own right. Then the loss and the model’s goal are different objects, which agree on the training distribution and can differ off it. That situation arises because capabilities are learned, which is the bitter lesson: the methods that scale are generic search and learning, so the goals of a capable system are also produced by learning rather than written in. Hubinger et al. (“Risks from learned optimization”, §4 below; their §2.1) push one level further: if diverse tasks favour general-purpose learned search over learned heuristics, the bitter lesson predicts models that are themselves optimizers.

Offer B for A plus \varepsilon/2: the agent accepts. Offer C for B plus \varepsilon/2, then A for C plus \varepsilon/2. The agent is back where it started, 3\varepsilon/2 poorer, and the trader can repeat the cycle as often as it likes. The failing axiom is transitivity: from A\prec B and B\prec C it would follow that A\prec C, contradicting C\prec A.

The argument says an agent that cannot be exploited this way has transitive preferences. Exploitation arguments for the other axioms are weaker and contested (completeness and independence are argued over; continuity has essentially none), so “non-exploitable implies expected-utility maximizer” is a heuristic, not a theorem. Even granting it, the conclusion is only that the preferences have some expected-utility representation. It says nothing about which U, or what U is defined over. Exercise 2.2 shows that the second point matters a lot.

Let U(h)=1 if every action in h is the one \pi would have taken given the history before it, and U(h)=0 otherwise. Following \pi gives U=1 with probability one, the largest possible value, so \pi maximizes expected U in every environment. (For a stochastic policy, use U(h)=1 when every action is in the support of \pi given the preceding history.)

So “is an expected-utility maximizer” is empty when U may be any function of the whole history. The claim behind classical alignment arguments is stronger: that capable agents behave as if maximizing a simple utility over states of the world, which is what produces instrumental convergence (§4). That is a substantive claim about which agents training produces, not a theorem.

Suppose T\vdash\Box\,\mathrm{safe}(b)\to\mathrm{safe}(b) for every b. By Löb’s theorem, T\vdash\mathrm{safe}(b) for every b, including actions that are in fact unsafe. So either T proves false things about safety, or the soundness statements are unprovable for every b whose safety T cannot already prove outright. (Taking \varphi to be a contradiction gives Gödel’s second incompleteness theorem as a special case.)

So an agent can’t approve a successor by trusting that the successor’s proofs are sound, if the successor uses the same theory. It can trust a successor that reasons in a strictly weaker theory T', if T proves \Box_{T'}\varphi\to\varphi. Naively, then, each self-modification loses some strength. Yudkowsky and Herreshoff build infinite descending families of theories, each still extending Peano arithmetic, and other partial workarounds exist, each with costs. This is the Löbian obstacle. It is a problem about agents that act on proofs, which is one reason it looked central when ideal agents were the model of AI and looks less central now.

(a) Fix s. Let f(a)=r(s,a)+\gamma\sum_{s'}P(s'\mid s,a)V(s') and g(a) the same with W. Then

|\mathcal TV(s)-\mathcal TW(s)|=\big|\max_a f(a)-\max_a g(a)\big|\le\max_a|f(a)-g(a)|=\gamma\max_a\Big|\sum_{s'}P(s'\mid s,a)\big(V(s')-W(s')\big)\Big|\le\gamma\|V-W\|_\infty .

The first inequality: if \max f is attained at a_1, then \max f-\max g\le f(a_1)-g(a_1), and symmetrically. Taking the maximum over s gives the claim.

(b) A \gamma-contraction on the complete space \mathbb R^{|S|} has exactly one fixed point (Banach); two fixed points V,W would satisfy \|V-W\|_\infty\le\gamma\|V-W\|_\infty, forcing V=W. Since V^*=\mathcal TV^*, \|V_k-V^*\|_\infty=\|\mathcal TV_{k-1}-\mathcal TV^*\|_\infty\le\gamma\|V_{k-1}-V^*\|_\infty, and induction gives the bound.

(c) With \gamma=1 the bound is \|\mathcal TV-\mathcal TW\|_\infty\le\|V-W\|_\infty, which is not a contraction, and in general there may be no fixed point (rewards that never stop accumulating) or many (with no terminal states, \mathcal T(V+c)=\mathcal TV+c, so adding a constant to a fixed point gives another). A finite horizon T rescues it: values are indexed by time, V^*_{T+1}\equiv0, and backward induction computes V^*_T,\dots,V^*_1 exactly in T steps. Episodic problems in which every policy reaches a terminal state are handled in the same spirit (Bertsekas’s “stochastic shortest path” theory).

(a) Given (s,a), the reward is r(s,a) and only s' is random, so

\mathbb E[y\mid s,a]=r(s,a)+\gamma\sum_{s'}P(s'\mid s,a)\max_{a'}Q(s',a')=(\mathcal TQ)(s,a).

So the Q-learning update is a Robbins–Monro stochastic approximation to the fixed point of \mathcal T, which is a \gamma-contraction on Q-tables by the argument of Exercise 3.1. With step sizes satisfying \sum\alpha=\infty, \sum\alpha^2<\infty and every (s,a) visited infinitely often, the table converges to Q^* (Watkins and Dayan, 1992), whatever behaviour policy generated the data.

(b) (\mathcal TV)(s) has the maximum outside the expectation. A single transition shows what happened after one action and says nothing about the actions not taken, so there is no unbiased one-sample estimate of the max over actions of an expectation. Without a model you can evaluate a fixed policy (TD(0)), but you cannot do the optimality backup for V. For Q, the action is already chosen and the maximum is over the next action, inside the expectation over s', where it can be read off the table. That is why model-free control learns Q.

(c) Larger, in general: \max is convex, so by Jensen \mathbb E\max_{a'}\hat Q(s',a')\ge\max_{a'}\mathbb E\hat Q(s',a')=\max_{a'}Q(s',a'), with strict inequality whenever the noise can change which action wins. Q-learning bootstraps on its own maxima, so the upward bias propagates and compounds: maximization bias. The standard fix is double Q-learning (van Hasselt, 2010; Double DQN, 2016): keep two estimates, choose the maximizing action with one and evaluate it with the other, so selection noise and evaluation noise are independent. The same mechanism (optimizing against a noisy estimate selects for its errors) is one cause of reward-model overoptimization (Gao et al., in §5); hacks such as editing a test exploit systematic gaps in the reward rather than noise.

(a) Improvement: MCTS. Starting from the network’s prior \mathbf p and value v, a lookahead search produces a move distribution \boldsymbol\pi that is usually stronger than \mathbf p, just as acting greedily on a lookahead improves on the current policy. Evaluation: the value head is regressed onto z, the outcome of games played by the improved policy, a Monte Carlo estimate of that policy’s value. The policy head is amortization: it is trained to imitate the search, \mathbf p\approx\boldsymbol\pi, so that next round’s search starts from a better prior and the improvement compounds. Tabular policy iteration can store the improved policy exactly; a network has to be taught it. Silver et al. call MCTS “a powerful policy improvement operator” and self-play with search “a powerful policy evaluation operator”.

(b) Inside the tree. Each simulation backs the leaf value up the path to the root, averaging values at each node (with signs alternating between the players). That is an approximate minimax backup, the two-player version of \max_a[r+V(s')], computed by sampling rather than full expansion. The network supplies the leaf values, so the tree is a Bellman backup bootstrapped from a learned V.

(c) Still hand-written in AlphaZero: the rules (a perfect simulator, and with it a Markov state), the reward (win, lose or draw), the input encoding, the network architecture and the search algorithm. MuZero learns the model, so it no longer needs the rules at search time, though the environment still provides them during training. The reward is supplied by hand in every system in the table. That is where the alignment sections pick up: learning took over the heuristic and then the model, but not the goal.

(a) The safe action has Q=0 under both backups. For the gamble,

Q_{\text{exact}}=\log\big(p\,e^{R}+(1-p)\,e^{-C}\big),\qquad Q_{\text{expected}}=pR-(1-p)C .

(b) With p=0.01, R=10, C=100: p\,e^{R}=0.01\,e^{10}\approx220.3 and (1-p)e^{-C}=0.99\,e^{-100}\approx3.7\times10^{-44}, which is negligible. So

Q_{\text{exact}}\approx\log220.3=10+\log0.01\approx5.39,\qquad Q_{\text{expected}}=0.1-99=-98.9 .

Exact inference prefers the gamble: the posterior policy is a softmax of Q, so the odds are e^{5.39}/e^{0}\approx220 to 1 in favour of the gamble. The expected backup rejects it; even a softmax policy built on it gives odds of about e^{-98.9}\approx10^{-43}.

(c) Using the hint,

p(\text{good}\mid s_t,\text{gamble},\mathcal O)=\frac{p\,e^{R}}{p\,e^{R}+(1-p)\,e^{-C}}=\frac{1}{1+99\,e^{-110}}\approx1-1.7\times10^{-46}.

Conditioned on “everything went well”, the agent has inferred that the gamble paid off. Exact inference tilts the dynamics as well as the actions, so the agent plans as if it could choose its luck.

(d) Exact inference prefers the gamble when p\,e^{R}+(1-p)e^{-C}>1. As C\to\infty this becomes R>\log(1/p), which does not depend on C at all: however large the catastrophe, any gamble with R>\log(1/p) looks good. The same caution applies to any scheme that conditions a model of the world on success. Levine §3 removes the problem by variational inference with the true dynamics held fixed (the agent may choose its actions but not its luck). The objective becomes maximum-entropy RL, the backup over next states becomes the ordinary \mathbb E\,V, and the gamble gets Q=-98.9.

(a) Expand the right-hand KL and substitute \log\pi^*=\log\pi_0+r-\log Z:

\mathrm{KL}(\pi\,\|\,\pi^*)=\sum_a\pi(a)\log\pi(a)-\sum_a\pi(a)\big[\log\pi_0(a)+r(a)-\log Z\big]=\mathrm{KL}(\pi\,\|\,\pi_0)-\mathbb E_\pi[r]+\log Z .

Rearranging gives the identity. Since \mathrm{KL}(\pi\|\pi^*)\ge0 with equality only at \pi=\pi^*, the maximum is \log Z=\log\mathbb E_{\pi_0}e^{r}, attained only at \pi^*. With a coefficient \lambda, divide the objective by \lambda: the maximizer is \pi^*\propto\pi_0e^{r/\lambda} and the optimal value is \lambda\log\mathbb E_{\pi_0}e^{r/\lambda}. As \lambda\to0, \pi^* concentrates on the reward-maximizing actions; as \lambda\to\infty it stays at \pi_0.

(b) \pi_0 is the prior, and e^{r} (or e^{r/\lambda}) is the likelihood of the “observation” that the action was optimal, as with p(\mathcal O=1\mid a)=e^{r(a)} in Exercise 3.4; \pi^* is the posterior. To make the likelihood a genuine probability, choose b\ge\max_a r(a) and set p(\mathcal O=1\mid a)=e^{r(a)-b}. This leaves \pi^* unchanged, and the evidence is p(\mathcal O=1)=e^{-b}Z, so the optimal value \log Z is the log-evidence up to the constant b.

So maximizing reward while staying KL-close to a reference policy has the form of a Bayesian update of that policy. This closed form is a one-step result: it applies to KL-penalized RL fine-tuning when each whole completion for a fixed prompt is treated as one action. Token-by-token, the same update holds with the reward replaced by a soft Q-function that includes the value of what follows.

(a) Optimization selects the achievable states that score highest. If some states in E are mapped by some \phi_i to high-utility states but are cheaper to reach than ordinary ones (a simulated diamond is easier to make than a real one), maximizing U\circ\phi_i selects them. So the optimizer is drawn to exactly the states where the translation matters. This is the ontological version of Goodhart’s law, and of perverse instantiation.

(b) Convergence and human-compatibility do the work. If the agent’s refined model contains a variable that matches the human “diamond”, that variable settles most of E: an exotic isotope of carbon in a diamond lattice is a diamond in the shared concept. What remains are cases where the human concept is itself undefined. Abstractability alone only says that some low-dimensional summary exists, not that it is ours. What is left over: natural abstractions are approximate and defined for typical states, while the cheapest high scorers are often atypical edge cases where even natural abstractions blur. So the hypothesis shrinks E but does not stop optimization from selecting states in it.

One reasonable set of answers; several entries are contested.