Overfitting

What Overfitting means

Overfitting is the failure in which a model learns the particular accidents of the examples it was trained on rather than the pattern that will recur, so its score on what it has seen keeps rising while its performance on anything new gets worse. Owning this changes what we ask of any explanation that fits perfectly. We stop being impressed by a fit that leaves nothing unexplained, because the data always carries noise, and a model with enough freedom will explain the noise too. We judge a model only on cases it has never seen, and we suspect every rule that was added to account for one awkward month. Before, the perfect back-test was the goal. After, it is the warning sign, and the honest number is the one from the held-out set.

Overfitting examples: where it shows up

• At work: a hiring rubric tuned to describe the last five successful hires in detail selects for their coincidences rather than for what made them good.
• With money: a strategy with dozens of rules, each added to fix one bad year, is a description of those years and a poor guide to the next.
• In the news: a diagnostic system that scored brilliantly at one hospital fails elsewhere, having learned which scanner the sickest patients were put through.

Where it sits

Open to walk. See where Overfitting sits on the sky →

Nearest on the sky

• Capabilities and Limits
• The Context Window
• Gradient Descent
• The Map Is Not the Territory
• Neural Networks
• Scaling Laws

A filled dot marks a star that is already open to walk.