ExpertsFMStop building AI wrappers. Start building agent harnesses.
The Startup Ideas Podcast · October 28, 2026
5 Selected Lessons:
heuristic · 3 framework · 2
Evals for objective tasks, A/B testing + hard-to-fake behavioral signals for subjective ones
Source
The Startup Ideas Podcast — Stop building AI wrappers. Start building agent harnesses. (October 28, 2026) · at 15:49 ▸
When this applies
when deciding how to evaluate a system/product whose output quality is partly objective and partly a matter of taste
The takeaway
Use hard-coded tests for the objectively verifiable slice of quality, and use large-scale behavioral A/B signals — especially ones costly to fake, like paid conversion — for the subjective slice.
Why it works
Evals work when there's a checkable ground truth (instruction-following, brand guideline adherence); taste and personal fit have no single ground truth, so you need population-level behavioral signals that reveal revealed preference instead of stated preference.
One level up
Measurement method should match the epistemic nature of the thing being measured: verifiable claims get tests, contested/subjective claims get revealed-preference data at scale.
Where it stops holding
This stops working when you lack a large, diverse user base to run statistically meaningful A/B tests, or when there's no monetization signal to serve as a hard-to-fake proxy.
Detail
procedure:
- Use evals (hard-coded test questions/benchmarks) for objective properties like instruction-following or brand-guideline adherence
- For subjective qualities (taste, personal fit), run large-scale A/B tests across the diverse user base
- Track editing time, sharing/export behavior as engagement signals
- Use free-to-paid conversion as the 'North Star' hard-to-fake signal
- Use downstream retention (weekly/monthly return) as a slower but equally hard-to-fake signal
benchmarks: {}
rationale: Paid conversion and retention are hard to fake because people only pay/return if they perceive real value.
In their words
Generally, people only pay if they're really seeing value out of a tool, so we use that one as a North Star.
Rapid model adoption testing loop
Source
The Startup Ideas Podcast — Stop building AI wrappers. Start building agent harnesses. (October 28, 2026) · at 24:11 ▸ · at 26:26 ▸
When this applies
when deciding how fast to integrate newly released AI models or tools into a live product
The takeaway
If you can run fast, low-cost live experiments, there's little reason to delay testing new capability — adopt a continuous testing cadence rather than periodic evaluation cycles.
Why it works
With a large existing user base, small live experiments can generate statistically meaningful signal within hours, making it cheap to test new models against the specific tasks in your harness's 'job ladder' (expensive-to-cheap tiers).
One level up
Infrastructure that shortens the feedback loop (fast experimentation at scale) converts uncertainty about new tools into near-immediate, cheap information.
Where it stops holding
This stops working when you lack enough live traffic to get meaningful signal quickly, or when the new model has no plausible fit with any existing harness task tier.
Detail
procedure:
- Map your harness's tasks into tiers by cost/capability needed (e.g., big model tasks vs. cheap/fast model tasks)
- On a new model release, immediately slot it into the matching tier as a test candidate
- Launch live experiments on a user sample rather than waiting for full evals
- Get usable data within hours given sufficient traffic
benchmarks: {"adoption_speed":"within the hour for interesting-class models (e.g., Anthropic/OpenAI releases)"}
rationale: Being first to adopt slightly better/cheaper models compounds into cost and quality advantages across hundreds of calls per task.
In their words
No, usually within the hour. If it's a model that I have reason to believe is going to be good.
Target prosumers and B2B professionals, not casual consumers, for paid AI tools
Source
The Startup Ideas Podcast — Stop building AI wrappers. Start building agent harnesses. (October 28, 2026) · at 18:40 ▸
When this applies
when deciding which customer segment to target for a subscription-based product
The takeaway
Subscription monetization works best where the buyer captures clear professional/economic value, not where usage is casual leisure.
Why it works
Work-related subscriptions succeed because the value is tied to professional output/income, whereas consumers resist subscriptions for low-stakes personal use (illustrated by password-sharing behavior); this mirrors the SaaS boom's reliance on business spending, not consumer spending.
One level up
Match your monetization model to where the value capture is concentrated, not where usage volume is highest.
Where it stops holding
This stops working when a product's professional use case can't be cleanly separated from casual consumer use, or when a segment already undervalues subscriptions despite work context.
Detail
rule: Focus product and monetization strategy on prosumers and B2B professionals rather than general consumers.
threshold: Referenced stat: only ~2% of consumers pay for AI tools, but the same data shows rapidly rising professional spend.
rationale: Work subscriptions historically drove the whole SaaS boom; consumers generally resist paying/subscribing for personal use.
In their words
And I think most people's main focus in AI is how to find those prosumers, the professional consumers, and how to break into B2B, where there's a totally different kind of spending going on.
Build on frontier model APIs first; fine-tune only after usage scale
Source
The Startup Ideas Podcast — Stop building AI wrappers. Start building agent harnesses. (October 28, 2026) · at 22:09 ▸
When this applies
when an early-stage startup debates building on third-party AI APIs versus training/fine-tuning proprietary models
The takeaway
Sequence infrastructure investment: use best-available third-party capability to win usage first, then invest in proprietary model work (fine-tuning, specialized models) once you have scale and identified gaps.
Why it works
Training custom models is capital-intensive and risks falling behind continuously improving frontier models; real moats come from accumulated usage data and specialized tasks the big models underperform on, which only emerge after scale.
One level up
Don't invest in defensibility mechanisms before you've proven there's something (usage, data) worth defending.
Where it stops holding
This stops working when there is a pre-existing cost, latency, or capability gap in frontier models so severe that no amount of prompting/context engineering can bridge it, even before you have scale.
Detail
rule: Early-stage startups should build on existing frontier models; fine-tuning/custom models become valuable later for cost reduction, personalization, and specialized tasks big models handle poorly.
threshold: Switch point: once you have 'real usage' and have identified tasks where big models consistently underperform (e.g., visual QA for Gamma).
rationale: Model capability keeps improving fast, so competing against the frontier directly is usually a losing early bet; usage is the main battle to win first.
In their words
I mean, let's be honest, just amassing real usage is the main battle people should be focusing on. By all means, use the models that are out there to get that usage in the first place.
Startup/experimentation success is an at-bats game, not a single breakthrough
Source
The Startup Ideas Podcast — Stop building AI wrappers. Start building agent harnesses. (October 28, 2026) · at 36:07 ▸
When this applies
when trying to improve any business metric through experimentation (marketing, product, outreach)
The takeaway
Treat growth/improvement as a volume game requiring many attempts rather than expecting any one test, post, or outreach attempt to be decisive.
Why it works
Most individual experiments fail or show marginal results; the cumulative volume of attempts, not any single test, is what statistically produces winning changes, similar to sales or dating success rates.
One level up
Skill built through repeated deliberate practice (a 'muscle') compounds, but only if you push through an initial period of low or no visible payoff.
Where it stops holding
This stops working when there's no feedback loop to learn from failed attempts, or when attempts are run without genuine variation (same mistake repeated isn't more 'at-bats').
Detail
rule: Expect to need many experiments before anything meaningfully moves the business; don't expect one big test to unlock success.
threshold: First experiment is expected not to work; a 'trough of sorrow' precedes visible results (compared to needing ~2 months of gym work before seeing a six-pack).
rationale: Experimentation (and social posting, blogging, cold outbound) all follow the same volume-based law of returns.
In their words
I think, uh, that's the hard part about being, uh, an experimenter is like, there's like a trough of sorrow that you have to go through.