Movement I · Where This Came From — Lesson 2
Significant Milestones in AI Advancement
A milestone matters because of the argument it settled, not the headline it made. Six results, and what each one actually proved.
Learning objective
By the end of this lesson you can name six turning points in AI and state, for each, the proposition it settled rather than the event that made the news — and you can explain why the through-line connecting them is the problem being attacked, not the method being used.
2.1 Milestones as proofs
Popular accounts of AI history are lists of things machines beat humans at. That framing is almost useless, because it invites the wrong inference — that progress is a ladder, each rung a slightly better version of the one below. Lesson 1 established that the field is at least two rival programmes rather than one lineage. This lesson gives you the tool for reading any individual result correctly.
Ask it of a benchmark score and the question usually has no good answer, which is itself informative — Lesson 7 shows what happens to a field when its benchmarks stop settling arguments. Ask it of the six results below and each has a crisp one.
2.2 The game-playing era
Games were AI's first proving ground for a practical reason: the rules are unambiguous, the outcome is unarguable, and a world champion exists to serve as the benchmark. Two matches, nineteen years apart, settled two entirely different questions.
Deep Blue, 1997 — search plus knowledge beats a human at a bounded problem
In May 1997 IBM's Deep Blue defeated the reigning world chess champion Garry Kasparov 3½–2½ in a six-game match in New York — a rematch, after Kasparov had won their 1996 encounter 4–2.1 Deep Blue was purpose-built hardware evaluating on the order of 200 million chess positions per second, combined with an evaluation function encoding a great deal of human chess knowledge.1
What it proved: for a problem with clear rules and a bounded search space, massive search plus encoded domain expertise is sufficient to exceed the best human. What it conspicuously did not prove is anything about intelligence in general. Deep Blue could not play draughts, hold a conversation, or explain a move. It is the high-water mark of the symbolic programme, and it is worth noting that the symbolic programme won this one outright.
AlphaGo, 2016 — learned intuition beats search where search cannot reach
Go had resisted the Deep Blue approach for two decades, for a countable reason: the game's branching factor makes exhaustive search hopeless, and no one had been able to write an evaluation function that could reliably say which of two Go positions was better. The knowledge required was exactly the kind Lesson 1 called tacit — strong players describe good positions as looking right.
DeepMind's AlphaGo combined deep neural networks with Monte Carlo tree search, learning position evaluation and move selection from data rather than from hand-written rules. It beat the European champion Fan Hui 5–0 in October 2015, reported in Nature in January 2016, and then defeated Lee Sedol 4–1 in Seoul in March 2016.2
What it proved: the evaluative judgement that resisted articulation could be learned from examples. This is Polanyi's paradox being routed around in public, on a stage, against a world champion. AlphaGo Zero, published the following year, went further and learned entirely through self-play with no human game records at all — establishing that even the human data was dispensable.3
Two boards, two arguments
A diptych. Left: a chessboard rendered as an exhaustively enumerated tree — thousands of thin branch lines fanning out and being evaluated, dense and mechanical, labelled search. Right: a Go board where the tree is barely sketched but the board itself glows in regions of "good shape," labelled learned evaluation. The visual argument is that the right-hand method looks at far less and knows far more. No faces, no humans — the contrast is between two methods, not between man and machine.
Aspect 16:9 · 3200×1800 px · Brand palette: navy #00040c, silver-blue #a8c0d8, column gold #a87848 · Style: schematic editorial diagram
2.3 The deep learning breakout
The 2012 ImageNet result is the least cinematic entry on this list and arguably the most consequential. Before it, computer vision was a discipline of hand-engineered features: researchers designed detectors for edges, corners, and textures, and the craft of the field lay in designing them well.
Krizhevsky, Sutskever, and Hinton entered the ImageNet Large Scale Visual Recognition Challenge with a deep convolutional network trained on GPUs, and took the top-5 error rate to 15.3% against 26.2% for the second-place entry.4 A gap that large in a mature competition is not an improvement; it is a different method winning.
What it proved: features learned from data beat features designed by experts — and the experts had been trying for thirty years. Every subsequent result on this list depends on the field having accepted that.
2.4 Transformers and scale
2017 — an architecture that parallelises
The 2017 paper Attention Is All You Need introduced the transformer.5 Its predecessors processed sequences one position at a time, which made them slow to train and forgetful over long distances. The transformer replaced that recurrence with attention: a mechanism that lets every position in a sequence look directly at every other position and weigh their relevance.
What it proved, and the reason it matters more than its accuracy numbers: sequence modelling could be made parallel. Recurrence forced computation into a chain; attention turned it into a matrix operation that GPUs could saturate. The architecture did not simply perform better — it made spending vastly more compute possible. Every model in this course sits on it.
Attention, without the mathematics
When the model processes the word it in “the trophy did not fit in the suitcase because it was too large,” attention is the mechanism that lets that position weigh every other word in the sentence and put most of its weight on trophy. No rule about pronouns was written by anyone. The weighting is learned from examples, and it is recomputed for every word in every sentence.
2020 — scale produces new behaviour, not just better numbers
GPT-3, described in Language Models are Few-Shot Learners, was a 175-billion- parameter transformer.6 The finding in the title is the milestone: the model could perform tasks it had never been trained on, given only a handful of examples in the prompt — and sometimes given only a description of the task.
What it proved: a sufficiently large model trained on a sufficiently general objective acquires capabilities that were never specified as training targets. Before this, adapting a model to a new task meant collecting task-specific data and retraining. After it, a great deal of adaptation happens in the prompt — which is why Lesson 3 spends its central chapter on how to write one.
November 2022 — capability becomes a product
The release of ChatGPT on 30 November 2022 introduced no new architecture and no new capability class. It changed the interface, and the interface changed everything downstream: adoption, investment, regulation, and public expectation. The date now functions as the index point in serious economic research on AI's labour-market effects — Lesson 11's central study measures from it.7
What it proved: distribution is a capability. The gap between a technology existing and a technology being used is itself a thing that can be closed, and closing it can matter more than the underlying advance.
| Year | Result | What it proved | Programme |
|---|---|---|---|
| 1997 | Deep Blue defeats Kasparov | Search plus encoded expertise beats the best human on a bounded problem | Symbolic |
| 2012 | AlexNet wins ILSVRC | Learned features beat expert-designed features | Connectionist |
| 2016 | AlphaGo defeats Lee Sedol | Judgement that resists articulation can be learned from examples | Hybrid |
| 2017 | The transformer | Sequence modelling can be parallelised — so scale becomes affordable | Connectionist |
| 2020 | GPT-3 and few-shot learning | Scale yields capabilities never specified as training targets | Connectionist |
| 2022 | ChatGPT released | Distribution is itself a capability | — |
2.5 What the list leaves out
Every milestone list is an argument about what counts, and this one is no exception. Three honest caveats belong on screen alongside it.
It is a list of demonstrations, not of deployments. The results here were chosen partly because they were legible — a match result, a competition score. Advances with no scoreboard, such as the steady improvement in machine translation through the 2010s, touched more lives and appear on no list.
It is skewed towards the winners. This is survivorship bias in its textbook form. The same period produced confident results that led nowhere. Lesson 1's winters are what that looks like at scale.
Recency is not yet history. The entries from 2020 onward are still being argued about by people with money riding on the answer. Treat the pre-2017 entries as settled and the rest as provisional — a discipline Lesson 13 turns into a forecasting method.
Demonstration protocol
- Choose one prompt that requires holding several constraints at once — a short rewriting task with three explicit requirements works well.
- Run it against a small, older, openly available model and against a current frontier model. Show both raw outputs side by side, unedited.
- Name what changed. Resist “it got smarter.” Point at the specific failure in the older output: a dropped constraint, a lost thread, an invented detail.
- Show the gap closing. Give the older model the same task broken into three separate steps. It often succeeds. The capability difference is partly a difference in how much structure the model can hold at once — which is Lesson 3's context window.
Key terms introduced here
- Transformer
- The neural network architecture introduced in 2017 that replaced sequential recurrence with attention, making sequence models parallelisable and therefore scalable.
- Attention
- The mechanism by which each position in a sequence weighs the relevance of every other position. Learned from data, recomputed for every input.
- Few-shot learning
- Performing a task the model was never trained on, given only a small number of examples supplied in the prompt.
- Parameters
- The numerical weights adjusted during training. GPT-3 had 175 billion; parameter count is a rough proxy for capacity and a poor proxy for capability.
- Survivorship bias
- The distortion introduced by studying only the results that succeeded. Every milestone list has it.
Evidence at a glance
The misconception to kill
“Each milestone was a bigger version of the last.”
Deep Blue and a modern language model share essentially no architecture. Deep Blue searched a tree of possibilities using hardware built for that single purpose and an evaluation function written by chess experts. A language model does no search of that kind at all. The through-line connecting these results is the problem being attacked — can a machine do this thing a person does? — not the method. Reading the list as a single ascending line predicts the future badly, because it implies the next step is more of the current one. Twice already, it has not been.
Self-check
Answer before revealing. Retrieval beats re-reading, and the feeling that re-reading is working is a known illusion.8
Deep Blue and AlphaGo both beat a world champion at a board game. Why are they not the same kind of result?
Deep Blue proved that search plus hand-encoded expertise suffices on a bounded problem; its knowledge was written by humans. AlphaGo proved that evaluative judgement nobody could write down could be learned from examples. Different question, different programme, different implication.
Why does the transformer matter more for what it enabled than for what it scored?
Because it removed the sequential bottleneck. Recurrence forced computation into a chain; attention made it a parallel matrix operation that hardware could saturate. That turned “spend far more compute” from impossible into merely expensive, and the scaling results of 2020 followed from it.
If ChatGPT introduced no new capability, why is its release date used as the index point in labour-market research?
Because economic effects follow use, not existence. The capability had been available through an API to a small technical population; the release made it available to essentially everyone. Researchers need a sharp date at which exposure changed, and that is the one.
Name one thing this milestone list is biased towards, and why that bias exists.
Legible, scoreable demonstrations — matches and competitions — because they produce an unarguable result on a date. Advances without a scoreboard, and advances that failed, are both underrepresented. The second is survivorship bias; Lesson 1's winters are the missing half of the record.
Practice log
1. Pick a recent AI announcement. What was the open question before it, and did it actually settle one?
2. Old model vs current model on the same prompt: name the specific failure in the older output.
3. Which entry on the list do you think will look least important in ten years, and why?
Endnotes
Source tiers: Primary peer-reviewed or archival; Institutional research body or government; Documentation first-party technical documentation; Journalism established newsroom; Trade vendor or aggregator — verify figures independently.
- Documentation IBM Research, “Deep Blue.” Deep Blue won the six-game 1997 rematch 3½–2½; Kasparov had won the 1996 match 4–2. IBM describes the system as evaluating roughly 200 million positions per second on purpose-built parallel hardware. ibm.com/history/deep-blue
- Primary D. Silver et al., “Mastering the Game of Go with Deep Neural Networks and Tree Search,” Nature 529 (2016): 484–489. The paper reports the 5–0 result against Fan Hui (October 2015); the 4–1 result against Lee Sedol followed in Seoul in March 2016. doi:10.1038/nature16961
- Primary D. Silver et al., “Mastering the Game of Go without Human Knowledge,” Nature 550 (2017): 354–359. AlphaGo Zero learned from self-play alone, with no human game records. doi:10.1038/nature24270
- Primary A. Krizhevsky, I. Sutskever, G. E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” NeurIPS 25 (2012): 1097–1105. Top-5 test error 15.3% versus 26.2% for the second-place entry in ILSVRC-2012. proceedings.neurips.cc
- Primary A. Vaswani et al., “Attention Is All You Need,” NeurIPS 30 (2017); arXiv:1706.03762. arxiv.org/abs/1706.03762
- Primary T. B. Brown et al., “Language Models are Few-Shot Learners,” arXiv:2005.14165 (2020); NeurIPS 33. The model described has 175 billion parameters. arxiv.org/abs/2005.14165
- Institutional E. Brynjolfsson, B. Chandar, R. Chen, Canaries in the Coal Mine? Six Facts about the Recent Employment Effects of Artificial Intelligence (Stanford Digital Economy Lab; revised 12 August 2026), which uses the ChatGPT release as its index date. Treated in full in Lesson 11. digitaleconomy.stanford.edu
- Primary H. L. Roediger III and J. D. Karpicke, “The Power of Testing Memory: Basic Research and Implications for Educational Practice,” Perspectives on Psychological Science 1, no. 3 (2006): 181–210. doi:10.1111/j.1745-6916.2006.00012.x
Bibliography
Primary research
- Brown, Tom B., et al. “Language Models are Few-Shot Learners.” arXiv:2005.14165 (2020). arxiv.org
- Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. “ImageNet Classification with Deep Convolutional Neural Networks.” NeurIPS 25 (2012): 1097–1105.
- Silver, David, et al. “Mastering the Game of Go with Deep Neural Networks and Tree Search.” Nature 529 (2016): 484–489. doi.org
- Silver, David, et al. “Mastering the Game of Go without Human Knowledge.” Nature 550 (2017): 354–359. doi.org
- Vaswani, Ashish, et al. “Attention Is All You Need.” NeurIPS 30 (2017). arxiv.org
Institutional research
- Brynjolfsson, Erik, Bharat Chandar, and Ruyu Chen. Canaries in the Coal Mine? Six Facts about the Recent Employment Effects of Artificial Intelligence. Stanford Digital Economy Lab, revised 12 August 2026. digitaleconomy.stanford.edu
Product and technical documentation
- IBM Research. “Deep Blue.” ibm.com
Learning science
- Roediger, Henry L., III, and Jeffrey D. Karpicke. “The Power of Testing Memory: Basic Research and Implications for Educational Practice.” Perspectives on Psychological Science 1, no. 3 (2006): 181–210.