ExpertsFMWhy Tejal Patwardhan stopped underestimating the models - Episode 21
OpenAI Podcast · June 16, 2026
5 Selected Lessons:
heuristic · 3 framework · 1 priority · 1
Realistic and long-horizon evals outrank static saturated ones
Source
OpenAI Podcast — Why Tejal Patwardhan stopped underestimating the models - Episode 21 (June 16, 2026) · at 17:16 ▸ · at 27:27 ▸
When this applies
when deciding which of several possible metrics deserves investment as the measured system's operating mode changes
The takeaway
Prioritize measurement methods that match the actual time-horizon and ambiguity of real usage over legacy, short, well-specified tests.
Why it works
Static, short-duration benchmarks cannot capture value delivered over days/weeks of autonomous work, so they systematically understate real capability and usefulness as task horizons lengthen.
One level up
As the operating envelope of a system expands, your measurement methodology must expand with it, or it will silently stop reflecting reality.
Where it stops holding
This stops working when tasks remain genuinely short/well-specified, in which case static benchmarks remain adequate.
Detail
hierarchy:
- Production usage / real-world task completion over long time horizons
- Realistic, ambiguous task evals (successor to GDPVal)
- Well-specified realistic task evals (GDPVal original)
- Static saturated academic benchmarks (lowest priority)
rationale: A static benchmark just doesn't measure the long-horizon nature of how these models can now work for days or weeks.
In their words
Like, these models can work for days or weeks for you. Internally in research, we've had the models just run for really long periods of time to do work. And one of the problems with an automated eval is you kind of need it to run within some amount of time and get results to be able to look at them.
Benchmark lifecycle: build, saturate, replace
Source
OpenAI Podcast — Why Tejal Patwardhan stopped underestimating the models - Episode 21 (June 16, 2026) · at 11:00 ▸ · at 14:01 ▸ · at 24:32 ▸ · at 31:42 ▸
When this applies
when any measurement system risks being outpaced by the thing it measures
The takeaway
Continuously replace saturated metrics with harder, more realistic ones that preserve the ability to distinguish performance, staying ahead of the pace of improvement.
Why it works
As the measured system improves, scores cluster near the ceiling, destroying the benchmark's discriminative power between genuinely different performance levels.
One level up
Any measurement instrument has a shelf life proportional to the rate of progress in what it measures; plan for its obsolescence from the start.
Where it stops holding
This stops working when the underlying capability has plateaued, in which case continual replacement adds cost without added discriminative value.
Detail
procedure:
- Start with the hardest available realistic test
- Track performance over model generations
- Watch for saturation (scores approaching 100%, models indistinguishable)
- Design a harder, more realistic successor benchmark
- Publish and repeat
benchmarks: {"saturation_signal":"models close to passing all questions correctly, near 100%"}
rationale: Once near-ceiling, you can't tell genuinely different intelligence levels apart, like comparing two geniuses on a high school exam.
In their words
Saturated is when a model is close to passing all of the questions correctly, like getting close to a 100% on the test. And once the benchmark is saturated, it's not super useful because you can't really tell models apart with that test.
Bench-maxing is bad
Source
OpenAI Podcast — Why Tejal Patwardhan stopped underestimating the models - Episode 21 (June 16, 2026) · at 11:00 ▸
When this applies
when any organization is tempted to optimize a visible metric instead of the underlying capability or value it's meant to proxy
The takeaway
Don't let the measurement become the target; invest in the real capability and let the measurement be a byproduct check.
Why it works
Optimizing directly for a measurable proxy diverts resources from the general improvements that actually satisfy the end user, creating a gap between score and real usefulness.
One level up
Goodhart's law applies to any metric used as both compass and destination — decouple scoring from resource allocation decisions.
Where it stops holding
This stops working when the metric is itself an unbiased, comprehensive proxy for the real goal with no gaming surface.
Detail
rule: Generally, benchmarking just to look good on evals (bench-maxing) is bad; invest compute in general model improvement, not eval appearance.
threshold: if you spend the large majority of budget on making evals look good rather than broad capability, you get a model that disappoints users
rationale: Users experience the real task, not the eval, so a mismatch produces bad user experience even with great eval scores.
In their words
Yeah. Bench maxing is, I would say, this idea that if someone training a model was just trying to look good on some evaluation or benchmark and not actually making the model generally useful, and I would say that's generally not super helpful
Guard against memorization and reward-hacking in eval design
Source
OpenAI Podcast — Why Tejal Patwardhan stopped underestimating the models - Episode 21 (June 16, 2026) · at 35:42 ▸
When this applies
when designing any test/metric for a system capable of learning the test instead of the underlying skill
The takeaway
Keep your evaluation data strictly out of training data, and stress-test your scoring criteria at scale for exploitable shortcuts before trusting the result.
Why it works
If the entity being measured has been exposed to the test answers (memorization) or can find a shortcut that satisfies the scoring rule without the intended capability (reward hacking), the score decouples from the true capability.
One level up
A measurement is only as trustworthy as its resistance to being gamed by the very thing it measures.
Where it stops holding
This stops working when the measured entity has no access to or ability to exploit the test's structure.
Detail
rule: Separate training data from eval data to avoid memorization; stress-test eval environments at scale to catch reward-hacking before trusting a 'not capable' result.
threshold: especially critical for safety testing, e.g. measuring cybersecurity capability, where false negatives are costly
rationale: Need to be sure before saying a model lacks a risky capability, since under-elicitation could mask true capability.
In their words
So that's one way to avoid that is to try to be really clean and disciplined about your data, not including any benchmarks or any evals that you want to measure, and that helps solve sort of the first problem that you laid out.
Build your own personal benchmark and re-test regularly
Source
OpenAI Podcast — Why Tejal Patwardhan stopped underestimating the models - Episode 21 (June 16, 2026) · at 29:33 ▸
When this applies
when evaluating any fast-improving tool/vendor where public reviews lag reality and your past judgment may be stale
The takeaway
Maintain your own small set of recurring tests against your real use case, and re-run them periodically rather than trusting a one-time past impression.
Why it works
Generic public benchmarks don't capture your specific use case, and capability changes every few weeks, so a personal recurring test is the only way to get calibrated, current signal.
One level up
In fast-moving domains, personal, repeated, direct experimentation beats relying on stale impressions or third-party benchmarks.
Where it stops holding
This stops working when the underlying system/market is stable and slow-moving, making repeated personal re-testing unnecessary overhead.
Detail
rule: Build personal benchmarks for your use case, and if a model fails something one week, retry it the next week rather than writing it off.
threshold: re-test every couple of weeks given how fast capability moves
rationale: People try something once, it fails, and they don't realize how fast things move — the best eval is to dogfood and keep using the model.
In their words
the best eval, honestly, is just to dog food or use the model. Like, people should just try to use the models as much as they can. And even if there are things that they think the model didn't do well one week, they should just try it again the next week.