How to Learn By Doing

14.1  Why reading about AI does not work

You have now read thirteen lessons. This one exists to tell you that reading them was not sufficient, and to say precisely why — because the reason is documented rather than motivational.

The skill being acquired is calibration: an accurate sense of where a system's competence ends. Calibration is knowledge about a boundary, and boundaries are learned by crossing them. No amount of description substitutes for the specific memory of having trusted an output that turned out to be wrong.

Two findings from the learning-science literature make this concrete, and both have already appeared in this course.

Retrieval beats review. Roediger and Karpicke showed that repeated testing produces substantially better long-term retention than repeated studying of the same material. On delayed tests measured in days or weeks, prior testing outperforms prior studying.1 Re-reading is better than nothing; it is considerably worse than being asked.

And we systematically misjudge which is working. In the same body of work, most students chose re-reading over self-testing, and the researchers named the effect the illusion of competence — fluent material feels learned, and feeling learned is not being learned.1 This is the same mechanism as the METR result in Lesson 10, where developers were 19% slower and estimated they had been 20% faster.2 Working with a fluent AI system is an almost perfectly designed environment for producing that illusion: the output reads like competence.

14.2  The loop

Four steps, repeated. The structure deliberately mirrors what the expertise literature identifies as the active ingredient in deliberate practice: immediate feedback, time for problem-solving and evaluation, and repeated performance to refine behaviour.3

Step 4 is the whole lesson. Without it you have an impression, and impressions are exactly what §14.1 shows to be unreliable — you will remember that it “worked pretty well,” which is the illusion of competence in its natural habitat. With it you have a record, and the record accumulates into calibration.

FIG-14-01

Two loops, one difference

Two cycles side by side. Left, "the loop most people run": three stations — pick task → do it with the model → feels fine — closing back on itself, drawn in flat grey, with a small ghosted label confidence grows, skill does not. Right, "the loop that works": four stations — pick a verifiable task → do it → check against ground truth → write down the failure — with the fourth station rendered solid in the accent gold and an arrow leading out of it to a growing stack of log pages that feeds back into station one. The visual argument is that the fourth station is what turns a circle into a spiral.

Aspect 16:9 · 3200×1800 px · Brand palette: navy #00040c, silver-blue #a8c0d8, column gold #a87848 · Style: process diagram

14.3  Choosing the first project

Three criteria. A project meeting all three will teach you more in two weeks than another thirteen lessons would.

1. You do it repeatedly. Repetition is what makes the feedback accumulate into a pattern rather than remaining a set of anecdotes.
2. You can tell good from bad instantly. This is Lesson 6's test — you must be able to verify faster than you could produce. Without it, step 3 of the loop silently stops happening.
3. Nothing breaks if it is wrong the first ten times. Learning requires a tolerance for error that high-stakes work does not have.

Worked examples across different kinds of work

The two failure patterns — both feel like good choices

14.4  Keeping the log

Four fields per entry. It should take ninety seconds, because a log that takes longer is a log you will stop keeping.

• The task — one line.
• What I expected — written before looking at the output. This field is what makes the log a calibration instrument rather than a diary.
• What actually happened — including successes; a log of only failures distorts in the opposite direction.
• What I will do differently — the transferable part.

Two disciplines make it work. Write the expectation first, or hindsight will quietly convert every result into the one you anticipated. And re-read the log monthly: the pattern across twenty entries is the calibration, and no single entry contains it.

Demonstration protocol

1. Show your own failure log. On screen, real entries, unedited.
2. Include something embarrassing. A case where you trusted an output and were wrong, ideally in front of someone else. This is the most persuasive sixty seconds in the entire course, and its persuasive power comes entirely from the discomfort.
3. Show the pattern. Read three entries that turned out to be the same failure in different clothes. That recognition is what calibration feels like from the inside.
4. Start an entry live for a task you have not tried, writing the expectation before running it. Let the audience see you be wrong in real time if that is what happens.