ExpertsFMRich Sutton and Khurram Javed: Why AI Models Stop Learning, and How to Start It Again
Training Data · August 18, 2026
5 Selected Lessons:
framework · 2 reasoning_pattern · 1 case · 1 heuristic · 1
The Bitter Lesson
Source
Training Data — Rich Sutton and Khurram Javed: Why AI Models Stop Learning, and How to Start It Again (August 18, 2026) · at 5:47 ▸ · at 18:01 ▸
When this applies
when building any system and deciding whether to encode expert knowledge directly or build a method that improves as more compute/resources become available
The takeaway
Favor general methods that scale with available resources over methods that encode current expert knowledge, because resources compound and knowledge doesn't.
Why it works
Methods that leverage increasing computation keep improving as compute grows, while methods built on fixed human knowledge hit a ceiling defined by that knowledge.
One level up
When choosing between a short-term edge and a long-term scalable trajectory, bet on the trajectory that compounds with an exogenously growing resource.
Where it stops holding
This stops working when the 'resource' that was supposed to keep growing (e.g., data, compute) actually hits a hard limit, as Sutton argues is happening with internet-scale data for LLMs.
Detail
procedure:
- Identify whether a proposed improvement relies on injecting human/domain knowledge or on a method that scales with computation (search, learning).
- Prefer the latter even if the former gives faster short-term gains.
- Don't treat prior knowledge and learning as enemies — use prior knowledge as a starting point, but ensure the system can keep learning beyond it.
benchmarks: {}
rationale: Computation/resources available to a method will keep growing over time; knowledge encoded by humans is static and eventually a ceiling.
In their words
It's all about not getting distracted by trying to put in your knowledge and just paying attention to what the problem needs and to how you can scale with computation.
Step-size optimization + generate-and-test to solve catastrophic forgetting
Source
Training Data — Rich Sutton and Khurram Javed: Why AI Models Stop Learning, and How to Start It Again (August 18, 2026) · at 38:53 ▸ · at 41:17 ▸
When this applies
when any incremental-update system risks destroying previously learned structure with each new piece of information
The takeaway
To add new knowledge without destroying old knowledge, separate 'which parts change' (meta-learned, heterogeneous plasticity) from 'how new capacity is created' (injected variation plus a fast test/selection mechanism), rather than updating everything uniformly via gradient steps.
Why it works
Giving every parameter its own learnable step size lets the system keep most weights nearly frozen (preserving old knowledge) while a few adapt quickly to new data; adding randomly-initialized new units (generate-and-test) lets the network grow new capacity without the slow, incremental drift of pure gradient descent.
One level up
Systems that must both preserve stability and remain adaptable need heterogeneous, targeted mechanisms for change rather than one uniform update rule applied everywhere.
Where it stops holding
This stops working when you try to retrofit it onto an already-trained model instead of training from scratch, since the step-size meta-learning and generate-and-test mechanisms must be learned jointly with the knowledge itself.
Detail
procedure:
- Give every weight its own step size, meta-learned so most become tiny (protecting old knowledge) and a few stay large (enabling fast adaptation).
- Periodically inject newly-initialized random units into the network (generate step) rather than relying solely on gradient-based growth.
- Let backprop act as the 'test' that selects which of the injected units prove useful.
- Train this combined system from scratch so it learns both the knowledge and how to learn, rather than retrofitting it onto an existing frozen model.
benchmarks: {}
rationale: Naive single-sample updates destroy previously learned structure across the whole network; targeted plasticity plus continual injection of variation avoids this while still allowing growth in complexity.
In their words
It is exactly like backprop, but you also plant new seeds of units that are newly initialized with random weights. Backprop only has random weights at the beginning of time, and then as you go on, all that randomness, all that variety from the randomness gets used up.
Big World Hypothesis as argument against synthetic data as a scaling solution
Source
Training Data — Rich Sutton and Khurram Javed: Why AI Models Stop Learning, and How to Start It Again (August 18, 2026) · at 9:45 ▸ · at 13:37 ▸
When this applies
when a team considers manufacturing proxy data/examples to overcome a finite real-world dataset, in any domain requiring expert judgment
The takeaway
A substitute for real experience that is authored by humans inherits the boundaries of human knowledge, so it cannot escape the bottleneck it was meant to solve.
Why it works
Any synthetic data generator is itself a small program written by humans, so it encodes only what humans already know; the real world is infinitely more complex than any such generator, so synthetic data can't substitute for experience in complex domains, and still requires human experts to judge quality.
One level up
Before adopting a workaround for a resource constraint, check whether the workaround secretly reintroduces the same constraint it claims to remove.
Where it stops holding
This stops working when the target task domain is simple/finite enough that a human-authored simulator can fully capture the relevant complexity (though even then, Sutton argues humans must still iterate the simulator using real-world feedback).
Detail
approach: Ask who decides what synthetic data is 'good' (answer: human experts), ask how complex the real task domain is (answer: infinitely complex), and ask whether the synthetic generator can be fixed without human intervention (answer: no, it needs humans to notice and patch the sim-to-real gap).
signals:
- A 'generator' of data is itself a small program bounded by its author's knowledge.
- Quality judgment of synthetic data requires domain experts — a human bottleneck.
- Self-driving sim-to-real gaps get fixed by human engineers noticing failures, not by the system learning on its own.
rationale: The big world hypothesis: the world is massively more complex than any agent's model of it, so any human-authored approximation (including a synthetic data generator) will be severely limited.
In their words
The big world is that the world is infinitely big. There are infinitely many things to learn. And you can have people generating these centric data sets, but there would always be more things to learn.
Moore's-law extrapolation to judge plausibility of an efficiency claim
Source
Training Data — Rich Sutton and Khurram Javed: Why AI Models Stop Learning, and How to Start It Again (August 18, 2026) · at 46:02 ▸
When this applies
when evaluating whether a future efficiency target is plausible given a known rate of technological improvement
The takeaway
Back-calculate a long-term projection to today's required starting point using the known compounding rate, and test whether that starting point is already plausible or achieved — this disciplines otherwise vague future claims.
Why it works
If compute/energy efficiency is improving at a known compounding rate (e.g., doubling every ~18 months), a multi-order-of-magnitude future target implies a specific, checkable efficiency level required today as the starting point.
One level up
Any multi-year prediction built on a compounding trend should be checked by working backward to what it implies must be true right now.
Where it stops holding
This stops working when the underlying compounding trend (e.g., Moore's law style doubling) itself slows, stalls, or doesn't apply to the specific resource (energy per parameter) being projected.
Detail
rule: To judge a long-horizon efficiency prediction, divide the target improvement by the expected number of doubling periods to find what must be achievable today.
threshold: Five to ten years at an ~18-month doubling cadence implies roughly two orders of magnitude of improvement, so a 20-watt target in 10 years implies ~2,000 watts should be achievable today.
rationale: So, five to ten years is two orders of magnitude of Moore's law. It's a standard improvement. If we double every 18 months, 10 years would give you two orders of magnitude.
In their words
So, five to ten years is two orders of magnitude of Moore's law. It's a standard improvement. If we double every 18 months, 10 years would give you two orders of magnitude.
Cursor's tab-autocomplete: batch updates as a limited form of continual learning
Source
Training Data — Rich Sutton and Khurram Javed: Why AI Models Stop Learning, and How to Start It Again (August 18, 2026) · at 22:07 ▸
When this applies
when a company tries to personalize a shared model using aggregated feedback from a large user base
The takeaway
Aggregating feedback across users is efficient for shared improvements but inherently inefficient for individualized adaptation.
Why it works
Batch updates from many users converge toward shared, population-level improvements but dilute any individual user's specific signal, since one update must satisfy everyone at once.
One level up
Shared learning infrastructure trades off individual responsiveness for collective efficiency; the right mechanism depends on whether the goal is population-level or individual-level adaptation.
Where it stops holding
This stops working when individual users have needs divergent enough from the population that batch-averaged updates fail to capture them.
Detail
situation: Cursor's tab-autocomplete model needs to improve from user interactions but updating on every single interaction individually would be unstable.
decision: They collect data from thousands/millions of users and perform one batch policy update at a time.
outcome: This works for shared, population-level improvement but is inefficient if a specific user wants to teach the model something specific to their own version.
lesson: Batch-based continual learning solves instability but at the cost of individualized adaptation — a real gap Oak Lab aims to close.
In their words
So, the way they do it, as far as I understand, is a lot of people are using tab. They collect all this data, so coming from millions of users or thousands of users, and then they do one update of the policy from this batch data.