ExpertsFMHow Warp ships 2,000 PRs a month with AI factories | Zach Lloyd (CEO, Warp)
How I AI · September 21, 2026
5 Selected Lessons:
heuristic · 3 framework · 2
Software factory as a closed-loop system of repos, agents, and automations
Source
How I AI — How Warp ships 2,000 PRs a month with AI factories | Zach Lloyd (CEO, Warp) (September 21, 2026) · at 0:00 ▸ · at 4:19 ▸ · at 8:09 ▸
When this applies
when trying to scale any knowledge-work process that currently depends on ad hoc individual tool use
The takeaway
Turn a scattered set of individual practices into a defined, codified system so that the system itself becomes observable and improvable, not just the outputs.
Why it works
Defining repos, integrations, and agents as code creates a closed-loop system where every task is recorded, enabling measurement and iterative self-improvement that individual local tool use cannot provide.
One level up
Infrastructure that captures its own operation is more valuable long-term than infrastructure that only produces output.
Where it stops holding
This stops working when tasks are too ad hoc or infrequent to justify building a persistent, codified pipeline around them.
Detail
procedure:
- Define repos, MCP servers, configuration, and agents all in code
- Allow work to be triggered from any integrated tool (Slack, Linear, GitHub) or automatically (e.g., crash reports)
- Route input through triage -> issue creation -> implementation -> PR -> QA (including computer-use video verification)
- Make the whole process public so others can observe or contribute
- Score every agent run and feed an observer agent to update the factory based on recurring failure modes
benchmarks: {}
rationale: Centralizing in the cloud and defining the workflow in code lets managers see velocity, cost, and quality across the whole team instead of per-individual black boxes.
In their words
it's an actual noun. It's like a product concept where it consists of a bunch of repos, a bunch of, like, MCP servers, like a bunch of configuration, and then a bunch of agents, essentially.
Replay past tasks under alternate model configurations to find cost/quality Pareto trade-offs
Source
How I AI — How Warp ships 2,000 PRs a month with AI factories | Zach Lloyd (CEO, Warp) (September 21, 2026) · at 28:32 ▸
When this applies
when choosing among alternative resources/providers for a repeatable task and wanting evidence-based trade-off data
The takeaway
Build your own benchmark from real historical tasks and your own scoring infrastructure to make resource-allocation decisions, rather than relying solely on generic external benchmarks.
Why it works
Replaying your own historical tasks with different configurations, scored by the same evaluation infrastructure used in production, produces evidence specific to your workload rather than relying on generic public benchmarks.
One level up
The most trustworthy optimization data comes from re-running your own real history under controlled variation, not from proxy benchmarks.
Where it stops holding
This stops working when you lack a reliable scoring/judge infrastructure to evaluate quality consistently across replays, making comparisons noisy or untrustworthy.
Detail
procedure:
- Curate a set of real past tasks (e.g., front-end tasks)
- Replay them with different model configurations
- Score outcomes using the same scoring infrastructure as production (LLM-as-judge, or human/algorithmic)
- Plot cost vs quality (Pareto chart) to see trade-offs
- Feed results into a model-routing strategy with evidence-backed choices
benchmarks: {}
rationale: Public benchmarks like SWE-bench are generic; recreating them on your own real data gives decision-relevant evidence for your specific workload.
In their words
The idea is you can just like there are these public benchmarks like SWE bench and terminal bench that are on, like, generic data, you can recreate on your own data, like from real past factory tasks, how would things have gone?
Human interactions per PR as a bottleneck metric
Source
How I AI — How Warp ships 2,000 PRs a month with AI factories | Zach Lloyd (CEO, Warp) (September 21, 2026) · at 12:20 ▸
When this applies
when trying to quantify how much manual correction/steering a semi-automated production process still requires
The takeaway
Measure the friction points (re-work, re-prompting, corrections) in an automated pipeline, not just throughput, to find where humans are actually constraining speed.
Why it works
Counting every re-prompt, comment, and correction across all integrated tools creates a proxy for how much human effort is still required to get agent output to an acceptable state, revealing hidden bottlenecks.
One level up
The bottleneck in automated systems is usually visible only if you measure correction cycles, not just completion counts.
Where it stops holding
This stops working when the organization lacks integrated tooling across all steps (chat, ticketing, code review) to actually capture every interaction.
Detail
rule: Track 'human interactions per PR' — re-prompts, comments, and corrections — as the main throughput-limiting metric.
threshold: No specific numeric threshold stated, but framed as analogous to 'how many times do you stop the line' in a factory.
rationale: The more times a human must steer or correct an agent, the more it limits overall throughput, similar to stopping a production line.
In their words
It includes all those things like how many re prompts in Slack, how many comments on linear, how many times did you have to correct the thing in code review. And so it's like a kind of like, yeah, it's a proxy for how much work did you have to do in order to, you know, get the thing to do the job.
Risk-score PRs to triage human review capacity
Source
How I AI — How Warp ships 2,000 PRs a month with AI factories | Zach Lloyd (CEO, Warp) (September 21, 2026) · at 14:53 ▸
When this applies
when allocating scarce expert attention across a large volume of automatically produced outputs
The takeaway
Reserve expensive human judgment for the tail of high-risk cases; automate clearance for the low-risk bulk.
Why it works
Automatically scoring the risk of each output lets low-risk items bypass costly human review, freeing reviewer bandwidth for the subset that actually carries meaningful risk.
One level up
Treat review/oversight as a risk-management allocation problem, not a uniform gate applied to everything.
Where it stops holding
This stops working when the risk-scoring model itself is unreliable or when the cost of an undetected high-risk error is catastrophic enough that even 'low-risk' misses are unacceptable.
Detail
rule: Risk-score every PR; auto-approve low/extra-low risk merges, require human review only for medium-high risk or outliers.
threshold: Low or extra-low risk score -> automatic approval; medium-high or outlier risk -> human review required.
rationale: This clears the bulk of low-stakes work automatically while preserving human review capacity for what actually matters.
In their words
I think code review becomes an exercise in risk management. I think that's right.
Minimum sample size before letting an observer agent self-correct the system
Source
How I AI — How Warp ships 2,000 PRs a month with AI factories | Zach Lloyd (CEO, Warp) (September 21, 2026) · at 0:00 ▸ · at 22:22 ▸
When this applies
when using any automated feedback loop to adjust a process based on observed failures
The takeaway
Require a meaningful sample size of failures before allowing automated or algorithmic correction, to avoid overcorrecting to noise.
Why it works
With too few failure examples, an automated corrective system will overfit to idiosyncratic single cases rather than genuine systemic issues.
One level up
Any self-improving/automated feedback loop needs a statistical floor of evidence to avoid chasing noise.
Where it stops holding
This stops working when failure types are extremely rare or heterogeneous, making it impractical to accumulate 20-25 similar failures before acting.
Detail
rule: Require roughly 20-25 failed runs as a 'real sample' before letting an observer agent create factory updates.
threshold: ~20-25 failed runs minimum
rationale: Otherwise, it starts to overcorrect based on single things.
In their words
You need like you need like a real sample, like, maybe twenty, twenty five failed runs, you need some significant sample size. Otherwise, it starts to overcorrect based on single based on single things.