ExpertsFMWhy I don’t think AGI is right around the corner
Dwarkesh Podcast · July 3, 2025
5 Selected Lessons:
case · 2 reasoning_pattern · 1 heuristic · 1 priority · 1
Weigh probability-distribution tails seriously even while holding a median forecast
Source
Dwarkesh Podcast — Why I don’t think AGI is right around the corner (July 3, 2025) · at 9:39 ▸
When this applies
when giving a point forecast for planning purposes, explicitly preserve and act on the tail scenarios rather than letting the median crowd them out
The takeaway
Stating a central estimate doesn't exempt you from preparing for plausible tail outcomes; good forecasting keeps both in view simultaneously.
Why it works
A wide probability distribution means low-probability-but-high-stakes scenarios (like a near-term misaligned superintelligence) still carry enough expected weight to justify preparation, independent of the median estimate.
One level up
Decision-relevant forecasting is about managing a distribution, not just picking a single number.
Where it stops holding
This stops working when resources for tail-scenario preparation trade off so heavily against median-scenario execution that hedging becomes self-defeating.
Detail
hierarchy:
- believe in and reason over the full probability distribution, not just a median
- preparing for plausible tail scenarios (e.g., misaligned 2028 ASI) is still rational even if your median is later
- give explicit 50/50 timeline bets as useful communication, while flagging they are not certainties
rationale: My probability distribution is super wide... work to prepare for a misaligned 2028 ASI still makes a ton of sense.
In their words
My probability distribution is super wide. And I want to emphasize that I do believe in probability distributions, which means that work to prepare for a misaligned 2028 ASI still makes a ton of sense.
Test transformative tech on your own narrow, favorable-case workflow before believing hype
Source
Dwarkesh Podcast — Why I don’t think AGI is right around the corner (July 3, 2025) · at 0:00 ▸
When this applies
when deciding whether a new technology is ready to replace human labor in your own operation
The takeaway
Before accepting a capability claim about a general-purpose technology, test it yourself on a task tailored to its strengths and measure the actual quality bar it clears.
Why it works
Hands-on use on tasks that should be the technology's easiest, most favorable case reveals real capability ceilings that marketing claims and anecdotes obscure.
One level up
Ground big claims about new technology in direct, repeated personal experimentation rather than secondhand accounts or theoretical capability.
Where it stops holding
This stops working when the task you pick to test isn't representative of the economically important use cases, or when you lack the domain expertise to judge quality.
Detail
approach: Use the AI repeatedly on self-contained, short-horizon tasks within its presumed strength zone (language in/language out), and rate output quality against a human benchmark.
signals:
- task is simple, self-contained, short horizon
- should be 'dead center' in the tool's repertoire
- measure against human performance baseline
rationale: If a tool scores only 5/10 on its best-case tasks, broader economic transformation claims are likely overstated.
In their words
I like to think that I'm AI forward here at the Thorcatch podcast and I probably spent on the order of a hundred hours trying to build these little LLM tools for my post production setup.
Use historical analogous algorithmic timelines as a base rate for 'simple-seeming' breakthroughs
Source
Dwarkesh Podcast — Why I don’t think AGI is right around the corner (July 3, 2025) · at 5:56 ▸
When this applies
when estimating time-to-breakthrough for a technically 'simple in retrospect' idea, anchor on how long comparable past ideas actually took to implement despite looking simple afterward
The takeaway
When an idea looks simple in hindsight, use the empirical time it took similar past 'simple' ideas to ship as your base rate, not your gut sense of its apparent simplicity.
Why it works
Retrospective simplicity of an algorithmic idea hides the real engineering, debugging, and pruning effort required to make it work reliably, so past comparable timelines are a better predictor than intuitive difficulty assessment.
One level up
Base rates from analogous past events beat intuitive difficulty judgments for forecasting technical timelines.
Where it stops holding
This stops working when the new problem is genuinely easier (less gnarly, more data available) than the reference case, which would make the base rate pessimistic rather than calibrated.
Detail
rule: If a simple-seeming idea in a comparatively easy domain took N years to implement, expect a harder domain with less data to take at least as long, likely longer.
threshold: ~2 years as the GPT-4-to-o1 reference point
rationale: Seeing how long it took to implement 'train on verifiable math and coding' implies we're underestimating the difficulty of the gnarlier computer-use problem.
In their words
Seeing how long it took to implement the idea, hey, let's train our model to solve verifiable math encoding makes me think that we're underestimating the difficulty of solving a much gnarlier problem of computer use, where you're operating on a totally different modality with much less data.
Saxophone-teaching analogy for why prompting isn't learning
Source
Dwarkesh Podcast — Why I don’t think AGI is right around the corner (July 3, 2025) · at 0:00 ▸
When this applies
when someone claims that written instructions/feedback can substitute for a learner directly practicing and adjusting in real time
The takeaway
Written instructions accumulated from others' failures cannot replace the learner's own iterative feedback loop for skills requiring embodied, tacit adjustment.
Why it works
Skill acquisition requires a tight feedback loop of attempt-observe-adjust by the same agent; relaying refined instructions to a fresh, unpracticed agent each time breaks that loop.
Where it stops holding
This stops working when the skill is purely declarative/textual, where instructions genuinely do transfer most of the needed information (e.g., simple fact lookup).
Detail
situation: Teaching saxophone normally means a student blows, listens, adjusts repeatedly.
decision: Imagine instead one attempt per student, with written notes passed to the next fresh student.
outcome: No student would learn to play Charlie Parker this way.
lesson: This is analogous to how LLMs are 'taught' via prompts/notes, which is why it fails to produce real skill accumulation.
In their words
How do you teach a kid to play a saxophone? Well, you have her try to blow into one and listen to how it sounds and then adjust.
50/50 timeline bets: small-business taxes by 2028, human-like on-the-job learning by 2032
Source
Dwarkesh Podcast — Why I don’t think AGI is right around the corner (July 3, 2025) · at 9:39 ▸
When this applies
when forecasting a capability milestone, anchor the estimate to an explicit analogous historical progression (e.g., GPT-2 to GPT-4 took 4 years) and state it as a 50/50 bet rather than a confident date
The takeaway
When forecasting a complex capability milestone, find the closest analogous past capability curve, use its known timespan as your estimate, and state your confidence level explicitly rather than implying certainty.
Why it works
Mapping the target capability onto a structurally similar past progression (language modeling maturation) gives a grounded multi-year estimate instead of an arbitrary guess, while labeling it as a 50/50 bet keeps appropriate uncertainty.
Where it stops holding
This stops working when the analogy breaks down (e.g., more compute/researchers now could accelerate progress, or a fundamentally different data/learning constraint makes the analogy invalid).
Detail
situation: Need to forecast when AI can do end-to-end small business taxes like a competent GM, and when AI can learn on the job as fast/organically as a human employee.
decision: Anchor computer-use forecast on the ~4-year GPT-2-to-GPT-4 analogy (predicting 2028); anchor continual-learning forecast on noting GPT-1 was 7 years ago and continual learning has no current path but 7 years is a long runway (predicting 2032).
outcome: Stated as 50/50 bets, not certainties.
lesson: Grounding forecasts in a historical analog capability curve, while explicitly flagging uncertainty, produces a more defensible timeline than either pure optimism or pessimism.
In their words
Preparing taxes for a small business fuels like for computer use, what GPT-four was for a language. And it took four years to get from GPT-two to GPT-four.