ExpertsFMRichard Sutton – Father of RL thinks LLMs are a dead end
Dwarkesh Podcast · September 26, 2025
5 Selected Lessons:
heuristic · 2 case · 1 priority · 1 reasoning_pattern · 1
Weak general methods (search + learning) beat strong human-knowledge-infused methods
Source
Dwarkesh Podcast — Richard Sutton – Father of RL thinks LLMs are a dead end (September 26, 2025) · at 40:27 ▸
When this applies
choosing, in any engineering field, between a solution built from general-purpose principles versus one built from domain-specific human expertise
The takeaway
Rank general, principle-based, scalable methods above expertise-laden, hand-crafted methods when judging what will ultimately perform best, even if the latter looks stronger in the short term.
Why it works
General methods (search, learning) scale with added compute/data without being bottlenecked by the limits of human-curated knowledge, so over a long enough horizon they overtake systems that depend on human-imbued expertise.
One level up
In any sufficiently long competition between generality-with-scale and specificity-with-expertise, bet on generality-with-scale.
Where it stops holding
This stops working on short timeframes or in narrow domains where there isn't enough scale/compute/data available yet for the general method to overtake the hand-crafted one.
Detail
hierarchy:
- general-purpose search and learning methods
- human-knowledge-imbued symbolic/expert systems
rationale: Historically called 'weak' because they lacked human-imbued power, these methods 'have just totally won' over decades, per Sutton's firsthand view of the field.
In their words
So those were called strong. And so I think the weak methods have just totally won. That's the biggest question from the old days AI. What would happen? And learning and search have just won the day.
Weighing a historical pattern (bitter lesson) against a genuinely novel future regime (multi-agent AI collaboration)
Source
Dwarkesh Podcast — Richard Sutton – Father of RL thinks LLMs are a dead end (September 26, 2025) · at 46:40 ▸ · at 49:47 ▸
When this applies
asking whether a historical rule-of-thumb derived from one era will still apply once the underlying structural conditions change (e.g., new kinds of actors, new coordination possibilities)
The takeaway
Treat empirically-derived rules of thumb as scoped to the conditions that produced them, and re-examine whether a structurally new regime (e.g., a new kind of actor or new coordination technology) changes which pattern applies.
Why it works
A historical generalization like the bitter lesson is itself just an empirical pattern from a specific 70-year period, not a law of nature; when a fundamentally new possibility arises (e.g., many copies of the same digital agent that can share/merge knowledge the way no biological population can), the old pattern's applicability should be re-examined rather than assumed.
One level up
Don't extrapolate historical patterns past the structural conditions that generated them; look for what's genuinely new that could break the pattern.
Where it stops holding
This stops working when the 'new' regime isn't actually structurally different enough to break the old pattern's underlying causes.
Detail
approach: Question whether a learned historical rule (bitter lesson) is a law or just an observation from a bounded period.
signals:
- new capability for digital agents to spawn copies, learn separately, and attempt to reincorporate knowledge centrally — something biological populations can't do
rationale: This opens a genuinely new strategic question (compute toward self vs. toward spawning copies) not covered by the old pattern.
In their words
The bitter lesson, oh, who cares about that? That's an empirical observation about a particular period in history. Seventy years in history, no longer doesn't necessarily have to apply the next seventy years.
Bitter Lesson: human-knowledge shortcuts eventually lose to scalable experience-based methods
Source
Dwarkesh Podcast — Richard Sutton – Father of RL thinks LLMs are a dead end (September 26, 2025) · at 7:59 ▸ · at 11:17 ▸
When this applies
when choosing between a solution that encodes expert/human knowledge directly versus one that scales with more raw data/compute and self-generated feedback
The takeaway
Methods that scale with compute/data without a knowledge ceiling will, in the long run, outcompete approaches reliant on hand-curated expertise, even if the latter wins early.
Why it works
Human-knowledge-based approaches feel good and work initially because they front-load useful priors, but they hit a data/knowledge ceiling; scalable methods that learn directly from experience keep improving with more compute and eventually overtake them.
One level up
Favor the approach whose growth is bounded only by resources you can keep adding, not by a finite store of existing knowledge.
Where it stops holding
This stops working when the 'experience' signal is too sparse, too slow, or too expensive to generate at scale, making the knowledge-based shortcut the only practical option for a long time.
Detail
rule: Every time researchers have started with human knowledge and then tried to add scalable computation, they've gotten their 'lunch eaten' by purely scalable, experience-based methods.
threshold: n/a (qualitative historical pattern)
rationale: People get psychologically locked into the human-knowledge approach, and it cannot scale as well as experience+computation.
In their words
But in fact and in practice, it has always turned out to be bad. Because people get locked into the human knowledge approach and they psychologically or now I'm speculating why it is. But this is what has always happened.
Generalization is not automatic in gradient descent — it must be engineered
Source
Dwarkesh Podcast — Richard Sutton – Father of RL thinks LLMs are a dead end (September 26, 2025) · at 35:03 ▸ · at 37:48 ▸
When this applies
assessing whether any optimization process (a trained model, a tuned process, an evolved system) that works on observed cases will transfer well to new, unseen cases
The takeaway
An optimization process will solve the problems it's shown, but whether that solution generalizes well or poorly to new situations is undetermined by the algorithm itself and depends on human-driven architecture/representation choices.
Why it works
Gradient descent (and similar optimizers) is only pressured to minimize loss on the data it sees; nothing in the objective itself favors representations that transfer well to new states, so good transfer only happens when humans iteratively adjust the setup until it does.
One level up
Don't credit an algorithm for 'generalizing well' — credit the designers who iterated the representation until good generalization happened to emerge.
Where it stops holding
This stops working if a domain has only one possible correct solution, in which case apparent 'generalization' is really just the unique answer being found, not evidence of a transfer mechanism.
Detail
rule: We don't have automated techniques that promote good generalization/transfer in modern deep learning; when it appears, it's because researchers manually sculpted representations until it worked.
threshold: catastrophic interference (new training wiping out old knowledge) as a diagnostic sign of bad generalization
rationale: Generalization just means training on one thing affects behavior on others — it can be good or bad, and nothing in the method ensures it's good.
In their words
Mean, gradient descent will not make you generalize well. It will make you solve the problem. It will not make you get new data. You generalize in a good way.
TD-Gammon to AlphaGo to AlphaZero: scaling + removing human priors
Source
Dwarkesh Podcast — Richard Sutton – Father of RL thinks LLMs are a dead end (September 26, 2025) · at 40:27 ▸ · at 43:57 ▸
When this applies
assessing whether a celebrated breakthrough in any field is a genuinely new method or an old method scaled up and stripped of unnecessary human-added assumptions
The takeaway
The next-generation improvement in a lineage of systems often comes not from adding more human-crafted knowledge but from stripping it out and relying more purely on the scalable, experience-driven method.
Why it works
AlphaGo scaled up temporal-difference-style learning (via its precursor TD-Gammon) plus better search; AlphaZero went further by removing human game-knowledge entirely and learning purely from self-play experience, and performed even better.
One level up
When a breakthrough looks sudden, check whether it's really an old general method scaled up and purified of ad hoc human input.
Where it stops holding
This stops working when the problem domain truly cannot generate enough self-play/experience data (e.g., real-world domains with few, expensive trials), making pure experience-based learning infeasible.
Detail
situation: Backgammon, then Go, needed AI systems to beat world-class human players.
decision: TD-Gammon used TD learning; AlphaGo scaled this with better search; AlphaZero removed human game knowledge and learned purely from experience via self-play.
outcome: AlphaZero outperformed AlphaGo, which itself had beaten world champions, validating that removing human priors and scaling pure experience-based learning kept improving performance.
lesson: Surprising breakthroughs in AI often validate 'simple basic principles' (search+learning) over hand-engineered, knowledge-heavy approaches.
In their words
AlphaZero used TD, and AlphaZero was applied to all the other games and did extremely well. I've always been very impressed by the way AlphaZero plays chess because I'm a chess player and it just sacrifices material for sort of positional advantages.