Humana ante oculos foede cum vita iaceret
in terris oppressa gravi sub religione,
quae caput a caeli regionibus ostendebat
horribili super aspectu mortalibus instans,
primum Graius homo mortalis tollere contra
est oculos ausus primusque obsistere contra;
quem neque fama deum nec fulmina nec minitanti
murmure compressit caelum, sed eo magis acrem
inritat animi virtutem, effringere ut arta
naturae primus portarum claustra cupiret.
ergo vivida vis animi pervicit et extra
processit longe flammantia moenia mundi
atque omne immensum peragravit mente animoque,
unde refert nobis victor quid possit oriri,
quid nequeat, finita potestas denique cuique
qua nam sit ratione atque alte terminus haerens.
quare religio pedibus subiecta vicissim
opteritur, nos exaequat victoria caelo.
— Lucretius, De rerum natura I.62–791
Course pages for the days taught by Charles R-W. Each notebook is a single page: pick what interests you, open the sections you want, and nobody is expected to finish everything. Math is typeset in the browser, so the first load needs a network connection.
CRW: These are readings and exercises I've compiled for the days I'm teaching at the Iliad Intensive. In all cases, they are far more than can be read in the allotted time. They are provided on the assumption that you will use your good judgement, and your LLMs, to determine what is worth your time.
Follow your talents, your curiosity, and your confusions. Please try to leave the day much more confused than you entered it. Consult me if you've any questions on the content, meta-questions about how to study it, or meta*-questions.
Click a day to open it.
古人の跡を求めず、古人の求めしところを求めよ
— 松尾芭蕉 (Matsuo Bashō)2
Read me for the format (keystone reading, then an annotated trail with exercises and folded solutions).
CRW: This day is about surprising and confusing things from deep learning. In previous iterations of the intensive, the main material was Zach Furman's discussion of the theoretical computer-science perspective on deep learning as program synthesis; you can find this in MDL3. I've added a lot more material because I know relatively little about Solomonoff/TCS and because I have many other beloved mysteries which I am excited to share.
There are roughly 4 different areas we cover:
- Deep learning in relation to traditional statistical learning theory [MDL1]: our sword shattered in our hands.
- Reflections on LLMs, pre-training, next-token-prediction, and text [MDL2]: why did all the 2010s labs start doing RL, and why did they switch?
- Deep learning as program synthesis [MDL3]: how to square the success of deep learning with TCS ideas like Solomonoff and Occam?
- Mysteries of deep learning [MDL4]: a potpourri of a hundred surprising things about how deep learning works in practice. Additional reflections on universality and interpretability. A mishmash, but the things I personally find most immediately salient. This is the lower bound on how much any good theory of deep learning must explain. Not a low bar.
MDL1 (stat-learning) is a tour through statistical learning, includes derivations etc. Textbook-like.
MDL2 is broadly about perspective building, and has little truly technical material.
MDL3 is Solomonoff and friends; not my cup of tea, you'll know if it's yours.
MDL4 is a tour through my favorite surprising observations from deep learning: things that make me feel confused, my 'bag of nails' to hit when I find hammers; there is no coherent story, only a steeping in confusion befitting of the most courageous of natural philosophers.
Interactive figures: double descent, spectral filters, SGD stationary spread vs posterior, superposition.
ἀεὶ ὁ θεὸς γεωμετρεῖ
— Plato, as reported by Plutarch3
Read me. Bayesian statistics in thermodynamic language, plus the learning coefficient.
In previous iterations of the intensive, this day was built around the original Iliad worksheet, Singular Learning Theory by Kai Ogden, Matthew Farrugia-Roberts and Zach Furman (with solutions, without), together with Furman's slides. Both are still here and still excellent.
The blog posts are my primary recommended reading: Liam Carroll's Distilling Singular Learning Theory (DSLT 0–4), Jesse Hoogland's Neural networks generalize because of this one weird trick and, with Stan van Wingerden, You're Measuring Model Complexity Wrong; and Zach Furman's Singular learning theory: exercises.
I've provided the notebooks below as a backup and a store of extra material: derivations, worked exercises, and code to check things. Each stands on its own, and you should be able to work through any of them. A separate write-up of mine on the effective-theory view is to come.
γηράσκω δ᾽ αἰεὶ πολλὰ διδασκόμενος
— Solon4
Read me. Each notebook starts with the paper to read; the annotated derivations and code are optional.
Strongly recommended: Central flows, the companion site to Cohen, Damian, Talwalkar, Kolter and Lee, Understanding optimization in deep learning with central flows (ICLR 2025). It explains the edge of stability and how to model it in three short parts: gradient descent, scalar RMSProp and RMSProp. It goes with TD2.3.
Read me. Influence functions, Bayesian influence functions and unrolling, derived.
The primary source for this day is Louis Jaburi's Iliad notes, Data (Attribution) for Alignment (web version, with solutions; PDF, without). In the session we work through his derivations together, and the derivation slides follow the notes section by section.
The notebooks, slides and videos below are a backup and a store of extra material: more derivations, worked exercises, and code to check things. They are adapted from Louis's notes (CC BY 4.0), with changes.
Narrated video lectures, standalone alternatives to the notebooks and slides:
Readings and exercises for Thursday 29 October, accompanying Eric Michaud and Brianna Grado-White's lectures. Start with section I of What if not Circuits?, then choose the topics that interest you.
1 “When human life lay foully prostrate before all eyes upon the earth, crushed beneath the weight of religion, which showed her head from the regions of heaven, glowering down on mortals with horrible aspect, a Greek was the first who dared to lift mortal eyes against her, and the first to stand against her. Neither the fame of the gods, nor thunderbolts, nor heaven with its menacing growl held him back; they only spurred the keen courage of his mind the more, so that he longed to be the first to break through the tight-barred gates of nature. And so the living force of his mind prevailed: he went out far beyond the flaming walls of the world and traversed the whole immeasurable all in mind and spirit, whence he returns to us in victory with the knowledge of what can come to be and what cannot, and by what principle each thing has its power bounded and its boundary-stone set deep. So religion in her turn is cast beneath our feet and trampled, and his victory raises us level with heaven.” (Lucretius, On the Nature of Things, Book I, lines 62–79, on Epicurus; translation ours.) ↩
2 “Do not seek to follow in the footsteps of the wise. Seek what they sought.” (More literally: do not seek the footprints of the ancients; seek what the ancients sought.) Attributed to Matsuo Bashō (from his farewell words to his student Kyoriku, 1693), after Kūkai. ↩
3 “God always geometrizes.” Plutarch, Table Talk (Quaestiones convivales) VIII.2, 718b–c, where the dinner guests debate what Plato meant by it. One of them grants that it is “nowhere clearly written in his books, but is well attested and is of the Platonic character.” ↩
4 “I grow old, always learning many things.” Solon, fr. 18 West, quoted by Plato (Laches 188b; Republic VII, 536d) and Plutarch (Life of Solon 2). ↩
Source and builders: github.com/CharlesR-W/crw-iliad-intensive-oct-2026.