ExpertsFMThe AI paradox: More automation, more humans, more work | Dan Shipper
Lenny's Podcast: Product | Career | Growth · May 24, 2026
5 Selected Lessons:
heuristic · 2 prediction · 1 case · 1 reasoning_pattern · 1
Benchmarks rise only on pre-framed, scorable problems
Source
Lenny's Podcast: Product | Career | Growth — The AI paradox: More automation, more humans, more work | Dan Shipper (May 24, 2026) · at 40:43 ▸ · at 44:19 ▸
When this applies
when interpreting any rising performance metric as evidence that a human role is becoming obsolete
The takeaway
Rising scores on a defined task measure commoditization of that task, not elimination of the judgment that defines which tasks matter.
Why it works
Benchmarks can only measure what has already been articulated and scored by humans; the act of noticing/framing a problem worth solving happens before and outside the benchmark, so saturating a benchmark doesn't prove the upstream human skill is replaced.
One level up
Separate 'can it execute a stated problem' from 'can it decide what the problem is' before concluding a role is obsolete.
Where it stops holding
This stops working if/when automated systems begin reliably generating their own problem framings without human prompting, collapsing the distinction Dan relies on.
Detail
approach: When a benchmark score rises, ask whether it reflects better execution of an already-articulated task, or genuine autonomous problem-framing.
signals:
- model follows literal instructions even when a bigger fix is warranted
- model patches around issues rather than proposing structural rewrites unprompted
- score gains concentrate on tasks humans already wrote down and could score
rationale: This is why Dan predicts coding models will 'still do this in a year' — take literal instructions rather than independently reframing — even as raw scores keep climbing.
In their words
benchmarks rise on problems that we've framed that we can articulate, that we can score. And there's a lot of work that's human work that it can't be scored until you write it down.
What distinguishes AI-generated 'slop' from acceptable AI-assisted work
Source
Lenny's Podcast: Product | Career | Growth — The AI paradox: More automation, more humans, more work | Dan Shipper (May 24, 2026) · at 1:03:09 ▸ · at 1:06:19 ▸
When this applies
when deciding whether any delegated/automated output is acceptable to put your name behind
The takeaway
Judge any delegated output not by who/what produced it, but by whether the accountable person actually understands and stands behind its content.
Why it works
Slop occurs when the producer invests less time/attention than the audience will spend consuming it and cannot defend its content; acceptable AI-assisted work occurs when the producer directed it carefully and can stand behind every line, even if they didn't type it.
One level up
Accountability, not authorship, is the real quality gate for delegated work.
Where it stops holding
This stops working for audiences that specifically value human authorship/craft as the point itself (e.g., literary writing, art), where even well-directed AI output may be inappropriate regardless of the producer's understanding.
Detail
rule: Accept AI-generated documents/emails as long as the sender fully understands and can defend every line; reject them if the sender clearly has no idea what's in it.
threshold: 'the slop one is it took them less time to make it than it takes me to read it, and they don't stand behind every line'
rationale: Enables Every to use AI-generated quarterly strategy reports and most emails while maintaining quality, and to label mixed human/AI authorship rather than ban AI writing outright.
In their words
The slop one is it took them less time to make it than it takes me to read it, and they don't stand behind every line. So, my expectation is if you send me an AI-generated document, I think that's great. And if we talk about it and it's clear, you have no idea what's in it, like big no-no, not allowed to do that.
Ride the model: playful re-testing of new releases against your own problems
Source
Lenny's Podcast: Product | Career | Growth — The AI paradox: More automation, more humans, more work | Dan Shipper (May 24, 2026) · at 1:16:19 ▸ · at 1:23:32 ▸
When this applies
when a new version of any powerful tool is released and you want to extract real advantage from it rather than fall behind
The takeaway
Maintain a standing personal test suite of your real problems and rerun it against every new tool release, treating failure as temporary rather than final.
Why it works
Capabilities that failed on a prior model version may silently succeed on a new one; routinely re-testing your hardest personal/work problems against each new release catches capability jumps the moment they arrive, rather than relying on general hype or benchmark headlines.
One level up
Durable advantage from a fast-improving technology comes from continuous curious re-probing, not from a one-time adoption decision.
Where it stops holding
This stops working if an organization blocks access to new models/tools, or if the person lacks the discretionary time/incentive to experiment outside assigned tasks.
Detail
rule: When a new model comes out, re-try the things it 'can't quite do yet' — keep 'turning the rock over' — applying it to whatever you care about, job or not.
threshold: Applied literally to his own senior engineer benchmark, which jumped from an effective 0/30 to 62/100 once GPT 5.5 was tried.
rationale: The only thing you need to do to avoid layoffs, per Dan, is 'ride the models' — use them for whatever you do and keep testing new capabilities as they emerge.
In their words
the way to ride the models is like not one specific thing because they're always changing, but it is to be curious and playful to apply the model, the new model, to whatever it is that you care about, whether that's your job or something outside of your job, and to keep turning over rocks because it may not work now, but it may work eventually.
SaaS tools will be used from inside agent harnesses, improving SaaS margins
Source
Lenny's Podcast: Product | Career | Growth — The AI paradox: More automation, more humans, more work | Dan Shipper (May 24, 2026) · at 21:17 ▸ · at 23:48 ▸ · at 36:26 ▸
When this applies
when any software/service provider faces pressure to bolt AI onto their product versus letting users bring their own AI
The takeaway
A platform shift that lets users bring their own compute/intelligence to your product can expand usage and protect margins simultaneously, rather than forcing a costly AI feature build.
Why it works
If a coding/work agent (Codex, Cloud Code) can browse and operate any website using the user's own AI tokens, SaaS companies don't need to pay for AI compute themselves — the user's agent supplies it, while usage (and thus demand) for the underlying SaaS product rises because agents make it easier to use SaaS a lot.
One level up
Before copying an entrant's playbook (build-your-own-agent) during a platform shift, check whether the shift structurally removes the cost you were trying to solve.
Where it stops holding
This stops working if agent harnesses add friction to direct SaaS use, if model companies restrict cross-site browsing (as Cloud Code currently does), or if the 'SaaS apocalypse' Dan dismisses turns out to be real for other reasons.
Detail
rule: Expect agents to increase SaaS usage and demand rather than replace SaaS products, because users bring their own AI tokens to SaaS rather than paying the vendor to add AI.
threshold: Stated as confident enough that 'I would buy SaaS stocks right now... not investment advice' and 'SaaS stocks will be up majorly in the next couple of years.'
rationale: Every's own SaaS spend is up year over year despite being AI-maximalist internally, supporting the 'agents increase users of SaaS' claim.
In their words
I think it saves your margin because now, what you're what the way that you're thinking then is not I have to build AI into this. It's more like I have to make a piece of software that humans and AI want to collaborate on together.
The Senior Engineer Benchmark
Source
Lenny's Podcast: Product | Career | Growth — The AI paradox: More automation, more humans, more work | Dan Shipper (May 24, 2026) · at 40:43 ▸ · at 44:19 ▸ · at 47:00 ▸
When this applies
when designing a benchmark to compare an automated system's performance to a skilled human's judgment
The takeaway
A model can look like it's closing the gap on a named competence while still lacking the un-scoreable, prior step of recognizing what the real problem is.
Why it works
Standard benchmarks score models on tasks framed and scored by humans in advance; they can't capture the human act of reframing a problem (e.g., 'this codebase needs a rewrite, not a patch'), so rising scores overstate real-world replacement of senior judgment.
One level up
Beware benchmarks that measure execution of a stated task; the harder, more valuable skill is deciding what the task should be.
Where it stops holding
This stops working as a distinguishing test once models reliably reframe problems unprompted rather than only executing literal instructions — which Dan expects to take meaningfully longer than raw task-execution benchmarks suggest.
Detail
situation: Dan's vibe-coded app Proof kept crashing after launch; two independent senior engineers rewrote the codebase from first principles, giving him ground truth to score against.
decision: He scores new models on the same prompt ('this is vibe-coded slop, rewrite it from first principles') and compares to the human baseline (high 80s/low 90s out of 100).
outcome: Models scored ~30/100 until GPT 5.5, which scored 62/100 — the first model willing to rip out old code rather than patch around it; still well below the senior-engineer baseline.
lesson: Even models that fix stated problems well usually don't independently decide a larger rewrite/reframe is needed — that's the differentiator that keeps senior engineers valuable.
In their words
And all the models until GPT 5.5 got like a 30 out of 100. And senior, like a human senior engineer gets like high 80s, low 90s out of 100. So there's a lot to go. And then I tried GPT 5.5 and it got like a 62.