Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken

Dwarkesh Podcast — Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken (May 22, 2025) · at 1:00:57 ▸

when allocating scarce specialist effort across multiple promising but unequally tractable opportunities

When ranking where to deploy limited expert effort, weight both (a) near-term tractability/value and (b) the team's intrinsic motivation/identification with the problem, and recognize that organizations are rarely fully optimized — they are making difficult prioritization trade-offs under pressure.

Coding offers both immediate economic value and tractable feedback loops, producing super-exponential returns as you approach solving it; computer use is harder to pose as a clean feedback loop and offers less resonance with researchers' own notion of 'smart.' Labs are resource-constrained, best-effort organizations, not perfect optimizing machines.

Assume any admired organization's capability gaps mostly reflect under-invested prioritization choices rather than fundamental ceilings, unless proven otherwise.

This stops working when the 'harder to pose as feedback loop' domain actually hits a structural ceiling rather than a prioritization gap — this remains an open empirical question per Sholto's own caveat.

Dwarkesh Podcast — Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken (May 22, 2025) · at 23:00 ▸ · at 1:45:38 ▸ · at 1:48:52 ▸

when building understanding of any opaque complex system from scratch, through successive representational breakthroughs

Understanding an opaque complex system progresses by first diagnosing the structural reason for illegibility, then building tools that relieve that structural constraint, then assembling the resulting legible units into causal chains.

Each stage solved the bottleneck of the previous: first diagnosing that models cram many concepts into few neurons (superposition), then giving the system enough 'representational space' to separate concepts cleanly (sparse autoencoders producing up to 30M features), then connecting those separated features across layers into causal chains (circuits) that explain multi-step reasoning.

Treat interpretability/understanding-building as an iterative engineering problem with its own research roadmap, not a one-shot insight.

This stops working when the underlying system's units of computation don't decompose into sparse, separable features even with more representational space — i.e., if superposition isn't the right diagnosis for all model behavior.

Dwarkesh Podcast — Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken (May 22, 2025) · at 0:00 ▸ · at 2:47 ▸

when deciding which process to invest scarce improvement effort in, prioritize the one where success/failure can be cleanly and cheaply verified

Improvement loops accelerate fastest in domains with cheap, objective verification; domains requiring subjective 'taste' judgments lag because the feedback signal itself is unreliable.

Clean, hard-to-game reward signals (does code compile, is the proof correct) let gradient-based improvement compound; fuzzy taste-based domains give noisy or gameable signal that doesn't reliably push capability up.

Before betting resources on improving a capability, ask whether you can cheaply and reliably tell success from failure — that answer predicts speed of progress more than raw difficulty.

This stops working when even seemingly verifiable rewards (like unit tests) can be hacked/gamed, so verification is only a short-term reliable differentiator unless adversarial robustness is also engineered.

Dwarkesh Podcast — Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken (May 22, 2025) · at 49:25 ▸ · at 52:50 ▸ · at 1:31:43 ▸

when designing any evaluation or quality-control process, check whether judging output is easier than producing it

A feedback loop scales only as far as the generator-verifier gap allows: easy-to-verify domains automate quickly, hard-to-verify ones (requiring taste/expertise) stay bottlenecked.

If an average person off the street could apply a grading rubric reliably, the model is probably capable of applying it too; but if grading itself requires deep expertise or taste, that bottleneck propagates into the training signal.

The scalability of any quality-control or training system is capped by how hard verification is, not how hard generation is.

This stops working when verification is cheap but systematically biased (e.g. human raters with length/sycophancy bias), producing a clean-looking but wrong signal.

Dwarkesh Podcast — Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken (May 22, 2025) · at 1:57:13 ▸ · at 2:02:11 ▸

when forecasting a technology's broad economic impact timeline based on trend extrapolation and economic incentive strength

A capability becomes 'almost guaranteed' when it only requires applying already-proven methods at larger scale/more data, and the economic payoff vastly exceeds the cost of doing so — removing the need for a conceptual breakthrough.

Current RL algorithms, even without further algorithmic breakthroughs, can automate hand-specified white-collar tasks given enough collected data; the economic payoff relative to total wage spend is so large that even brute-force, manually-scaffolded automation is worthwhile.

Distinguish forecasts that depend on uncertain future breakthroughs from forecasts that only require scaling known methods against a sufficiently lucrative incentive — the latter deserve much higher confidence.

This stops working if data collection for the long-tail of white-collar tasks proves far harder/slower than expected, or if algorithmic progress on long-horizon tasks actually does stall (which Sholto says he doesn't believe is happening).