ExpertsFMWhy PMs are suddenly valuable again | Tulsee Doshi (Head of Product, Gemini)
Lenny's Podcast: Product | Career | Growth · November 8, 2026
5 Selected Lessons:
framework · 2 heuristic · 2 reasoning_pattern · 1
Start evals small — five-to-twenty prompts before scaling
Source
Lenny's Podcast: Product | Career | Growth — Why PMs are suddenly valuable again | Tulsee Doshi (Head of Product, Gemini) (November 8, 2026) · at 21:25 ▸
When this applies
when you don't know how to start measuring quality for a new, ambiguous capability
The takeaway
When building any new measurement system for a fuzzy quality, start with a minimal self-rated sample to define 'good' before investing in scale or automation.
Why it works
Starting with a tiny, self-rated set of examples forces you to clarify your own definition of 'good' before worrying about scale or rigor, which is the actual hard part; scale can be added once that clarity exists.
One level up
Clarity of judgment should precede scale of measurement, not the other way around.
Where it stops holding
This stops working when the quality bar is already well-defined and agreed upon, in which case jumping straight to large-scale automated evaluation is fine.
Detail
procedure:
- Start with just five prompts stress-testing the capability you care about
- Scale to roughly 20 prompts where you can see signal between models A and B
- Act as your own rater first to clarify what 'good' means before formalizing
- Once clarity exists, figure out how to scale the evaluation
benchmarks: {"initial_set":"5 prompts","mini_eval":"~20 prompts"}
rationale: Identifying what 'good' looks like, especially for long multi-turn trajectories, is the hard part — small scale removes the barrier to starting.
In their words
I think for me, a big thing is start with even just five prompts, right? Like start, you don't have to start large.
The 60-20-20 rule for building AI products ahead of the model curve
Source
Lenny's Podcast: Product | Career | Growth — Why PMs are suddenly valuable again | Tulsee Doshi (Head of Product, Gemini) (November 8, 2026) · at 33:14 ▸
When this applies
when building a product on top of a rapidly improving underlying technology whose future capability is uncertain
The takeaway
When building on a fast-improving platform, allocate your build across three tiers — what works now, what's nearly there, and what's aspirational — rather than either building only for today's limits or betting entirely on a future capability you can't precisely predict.
Why it works
Splitting the product into a fully-working core, a near-future layer that's imperfect-but-usable, and an aspirational layer that only shows 'glimmers of light' lets the product work today while positioning it to automatically improve as the underlying technology improves, instead of betting the whole product on an uncertain future capability.
One level up
Treat capability-timing uncertainty as a portfolio allocation problem, not a single point prediction to get right.
Where it stops holding
This stops working when the underlying technology's improvement is too slow, stalled, or unpredictable in direction, making the 20%/20% 'glimmers of light' never materialize, or when the product is simple enough that traditional MVP thinking fully suffices.
Detail
procedure:
- Build 60% of the product on capabilities that fully work today
- Build 20% on capabilities that work well enough now, with visible path to near-future improvement
- Build 20% aspirationally, where you see only glimmers of capability but a vision of what would be amazing if cracked
- Stay willing to retest and rebuild the harness as models improve rather than locking into rigid structures built around older model limits
benchmarks: {"working_now":"60%","near_future":"20%","aspirational":"20%"}
rationale: Models change so fast that building purely for today's MVP wastes the opportunity to ride improvement, while betting purely on a 3-month-out prediction is unreliable even for insiders.
In their words
I think what you want to do is actually build such that like 60% of what you've built works now. And your prototype is at least functional, right? Then there's a set of maybe the 20% where you're like, it's not perfect, but it like works well enough.
Models are grown, not built — PM job is to shape emergent behavior after the fact
Source
Lenny's Podcast: Product | Career | Growth — Why PMs are suddenly valuable again | Tulsee Doshi (Head of Product, Gemini) (November 8, 2026) · at 0:00 ▸ · at 2:03 ▸
When this applies
when managing any complex system whose outputs can't be fully specified in advance — only nudged toward and then interpreted
The takeaway
When a system's output can't be fully predicted from its inputs, the value-creating work shifts from upfront specification to real-time observation and post-hoc shaping of what the system surprisingly produces.
Why it works
Training produces emergent, non-deterministic results even when inputs and goals are carefully designed, so the PM's real leverage comes after the fact — observing what actually emerged and shaping it into value, not just specifying inputs.
One level up
Treat intentional design and emergent discovery as two separate, sequential disciplines rather than assuming design alone guarantees the outcome.
Where it stops holding
This stops working when the system is actually deterministic and fully specifiable, in which case upfront specification alone suffices and post-hoc shaping adds no value.
Detail
approach: Set intentional goals and evals for the model, but expect unpredictable 'magic' in outputs; actively hunt for where the model uniquely shines and then intentionally shape that into user-facing value.
signals:
- model behaves unexpectedly well or differently on a capability
- customers ask for predictable goal delivery but reality diverges
- researchers track metrics climbing during a run but final behavior still surprises
rationale: Because training produces emergent results, the PM's job splits into (1) intentional goal-setting and (2) post-hoc discovery and shaping of what actually emerged.
In their words
It's kind of like Snape's potions class. You can put all the ingredients in, but you also have to expect that there will be a little bit of magic, and that magic is going to be surprising.
Evaluate end-to-end, not just the model in isolation
Source
Lenny's Podcast: Product | Career | Growth — Why PMs are suddenly valuable again | Tulsee Doshi (Head of Product, Gemini) (November 8, 2026) · at 28:37 ▸
When this applies
when judging the quality of any component that will actually be experienced through a larger pipeline or delivery system
The takeaway
Evaluate the full delivered experience — including judge quality and downstream serving/processing — not just the core component in a sandboxed test, because intermediate layers can silently distort perceived quality.
Why it works
Production mitigations, filters, and serving infrastructure can alter the experienced quality of an underlying model even if the model itself tests well in isolation, and the quality of the judge rating the output also caps measured accuracy.
One level up
The unit of evaluation should match the unit of actual user experience, not the unit that's easiest to isolate and test.
Where it stops holding
This stops working when there is no meaningful gap between the isolated component and the deployed experience, making end-to-end testing redundant overhead.
Detail
rule: Check that (a) the judge evaluating the model is itself high quality, and (b) the production stack — mitigations, filters, serving — isn't distorting the measured output versus what users actually experience.
threshold: Applies especially to nuanced qualities like voice naturalness/tenor where serving infrastructure can degrade quality invisibly.
rationale: An inaccurate judge or a lossy serving pipeline can make a genuinely good model look bad (or vice versa) in evaluation.
In their words
Whether you evaluate on your production stack with all of your mitigations and filters might affect your model performance. Right. How your model is served might impact your model performance.
Burnout comes from chaos and unenjoyable grunt work, not hours alone
Source
Lenny's Podcast: Product | Career | Growth — Why PMs are suddenly valuable again | Tulsee Doshi (Head of Product, Gemini) (November 8, 2026) · at 44:09 ▸
When this applies
when managing a team working extremely long hours and trying to prevent burnout
The takeaway
To manage burnout in a high-intensity environment, target the sources of churn and unenjoyable low-value work rather than simply trying to reduce total hours worked.
Why it works
Burnout arises from churn/chaos and from spending time on work that feels meaningless or wasteful, not from total hours worked per se; removing the chaos and the grunt-work components — even while hours stay high — reduces burnout because the remaining work is engaging.
One level up
Treat the quality/composition of effort, not just its quantity, as the lever that determines sustainability.
Where it stops holding
This stops working when hours become so extreme that even meaningful, well-organized work still causes physical/mental exhaustion regardless of enjoyment.
Detail
rule: Burnout comes from (1) feeling churn and chaos and (2) doing work you don't enjoy or that feels like grunt work/waste of time — not from hours alone.
rationale: Work you're passionate about, even if exhausting, brings joy rather than burnout; AI tools can help by removing grunt work and freeing time for creative, meaningful work.
In their words
I think burnout comes not just from working a lot. It comes from, I think, two places. I think it comes, one, from when you feel like there is churn and chaos. I think that introduces burnout.