Source
Y Combinator Startup Podcast — What Happens When Everyone Can Build Software (October 30, 2026) ·
at 32:16 ▸ ·
at 35:47 ▸
When this applies
when a system's performance depends on an internal configuration (indexing, structure) that's expensive to hand-tune but measurable
The takeaway
Convert any optimization problem with a measurable objective into a closed loop where an agent proposes changes, evals grade them, and only improvements get merged — a primitive form of recursive self-improvement for a narrow domain.
Why it works
By creating an eval dataset from real production traces and categorizing retrieval tasks, you can let an agent run in a loop that proposes changes, measures them against the evals, ships improvements automatically, and logs failed attempts — turning tuning into a continuous automated process instead of manual engineering.
One level up
Any well-defined, measurable subproblem can be handed to an agent-in-a-loop system once you have good evals; the eval suite becomes the actual asset, not the hand-written logic.
Where it stops holding
This stops working when you lack a reliable eval/metric for 'better,' since the loop has no signal to optimize against and can silently regress or overfit.
Detail
situation: GBrain needed to retrieve the right few pages from a half-million-document personal knowledge base efficiently and cheaply for agents.
decision: Built a separate 'GBrain evals' repo with ~40 categories of retrieval tasks (efficiency, cost, latency) derived from production traces, and set a persistent 'captain' agent to continuously auto-research, test, and ship improvements, logging failures as markdown notes.
outcome: Carrie describes it as 'a very, very primitive version of RSI for a particular use case,' continuously improving retrieval scores without manual re-engineering.
lesson: A narrow, well-evaluated domain can be handed over to a continuous agent-improvement loop, and this generalizes to 'anyone's retrieval use case' by swapping the data/traces.
In their words
And then the cool thing about evals is then you can just tell the agent, like loop, you know, just auto-research and loop. And anything that improves your stats on this eval, like ship it into master on by default.