ExpertsFMNeil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]
Invest Like the Best with Patrick O'Shaughnessy · August 25, 2026
5 Selected Lessons:
framework · 2 reasoning_pattern · 1 heuristic · 1 case · 1
Throughput vs. latency trade-off and choosing which side to build for
Source
Invest Like the Best with Patrick O'Shaughnessy — Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488] (August 25, 2026) · at 18:13 ▸ · at 22:33 ▸
When this applies
when designing any system that must serve many users/requests and must decide whether to optimize for individual response speed or aggregate volume/cost
The takeaway
Any shared-resource system forces a choice between serving few things very fast (low latency, lower utilization) or many things cheaply (high throughput, higher per-item wait); you cannot maximize both simultaneously on the same hardware configuration.
Why it works
Batching many requests together maximizes utilization of parallel compute units (lower cost per unit) but increases the time any individual request spends in the system; narrow-fast and wide-slow are the only two shapes a shared resource can take.
One level up
Pick which half of a fundamental, unbreakable trade-off your customer segment actually needs, rather than trying to be best at both.
Where it stops holding
This stops working when customers have strict mixed requirements (both fast AND cheap) that cannot tolerate a trade-off, forcing a more expensive hybrid architecture.
Detail
procedure:
- Identify whether your customer needs minimum latency per request or maximum cost efficiency across many requests
- If latency-first, use NVLink/tensor parallelism across many GPUs (sublinear speedup, higher cost)
- If throughput-first, use expert/pipeline parallelism, batch heavily, and hide communication latency
- Accept degraded single-request latency (bus analogy) in exchange for much lower per-token cost
benchmarks: {"sublinear_nvlink_scaling":"8x hardware yields ~4-5x speed, not 8x"}
rationale: GPUs are throughput machines; forcing them into a low-latency chatbot shape (today's dominant usage) fights their natural happy path.
In their words
There's a fundamental trade-off on the GPU between being throughput-oriented or latency-optimized. And everyone has chosen latency optimization because the shape of usage was chatbot-oriented.
Hiring for teachable skill vs. unteachable trait (love of performance)
Source
Invest Like the Best with Patrick O'Shaughnessy — Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488] (August 25, 2026) · at 1:02:01 ▸
When this applies
when hiring for a technical role in a domain whose specific tools/technologies are changing so fast that deep tool-specific experience quickly becomes obsolete
The takeaway
In fast-moving technical domains, hire for durable intrinsic traits that predict ability to learn whatever tool comes next, not for current tool-specific experience which will depreciate.
Why it works
Specific tool experience (CUDA years, AI experience) decays in value as the underlying tools evolve rapidly, but an intrinsic trait like curiosity/love of deep understanding compounds regardless of which tool is current, and cannot be instilled through training.
One level up
When the specific skillset is a moving target, optimize hiring for the meta-skill (curiosity, drive) that lets someone re-skill continuously.
Where it stops holding
This stops working in domains where tools are stable for long periods, making specific experience genuinely durable and valuable.
Detail
procedure:
- Do not screen primarily for years of CUDA/AI experience — it depreciates too fast
- Screen for curiosity and 'love for performance' — willingness to dig into microsecond-level machine behavior
- Favor people who are both good students and good teachers (e.g., former TAs)
- Build a collegial culture of whiteboard sessions where knowledge flows both ways
rationale: CUDA/GPU programming has evolved so much in five years that deep past experience is a 'huge red herring'; curiosity is the one trait that can't be taught.
In their words
I don't look for lots of AI experience. I don't look for CUDA experience at all. That's actually a huge red herring.
Find the unserved segment of a market everyone else is optimizing against
Source
Invest Like the Best with Patrick O'Shaughnessy — Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488] (August 25, 2026) · at 4:20 ▸ · at 7:25 ▸
When this applies
when an entire industry has converged on optimizing one variable (speed, size, etc.) because one dominant customer pulled it that direction
The takeaway
When an entire competitive set has been shaped by one influential early customer's requirements, look for the emerging requirement that customer doesn't have.
Why it works
Early market leaders get pulled toward the loudest, earliest customer's needs; this creates a structural blind spot for a differently-shaped need (throughput over latency) that only becomes visible as usage patterns shift.
One level up
Competitive openings often appear not where nobody is building, but where everybody is building the same thing for the same reason.
Where it stops holding
This stops working when the dominant requirement never actually shifts, or shifts back, leaving the new segment too small to build a sustainable business on.
Detail
approach: Identify the customer who shaped the market's current optimization target, then ask what happens when usage patterns move away from that customer's needs.
signals:
- one customer dominates early growth narrative
- all competitors converge on the same metric (latency)
- emerging use cases (agents) have different tolerance curves
rationale: The gap exists because incumbents are locked into serving the loudest early need, not because the new need is unknowable.
In their words
The challenge is all those companies, you could take your pick, base 10, fireworks together, they all focus on low latency inference. And they were pulled in that direction by one very important customer, Cursor.
Speed of light / absolute performance target
Source
Invest Like the Best with Patrick O'Shaughnessy — Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488] (August 25, 2026) · at 18:13 ▸
When this applies
when optimizing any engineered system's efficiency
The takeaway
Measure performance against the absolute physical/theoretical limit, not against relative standing versus rivals.
Why it works
Benchmarking against the theoretical physical maximum, rather than against competitors, prevents complacency once you're 'ahead' and keeps pushing toward real ceilings.
One level up
Comparing yourself to competitors caps your ambition at 'better than them'; comparing to the theoretical limit has no cap.
Where it stops holding
This stops working when the theoretical limit is unknown or unmeasurable, making the target abstract and unmotivating for teams.
Detail
rule: Chase 100% of the hardware's theoretical 'speed of light' performance, not relative numbers versus competitors.
threshold: 100% of theoretical peak throughput/compute given a chip's specs
rationale: NVIDIA's culture of chasing speed of light produced extremely retained, motivated engineers and genuinely squeezed performance out of each chip generation.
In their words
To this day, I tell all my engineers, we're chasing 100% speed of light. I don't care about relative numbers versus the competition. I only care about absolute numbers.
NVIDIA's deliberate allocation strategy to avoid customer power concentration
Source
Invest Like the Best with Patrick O'Shaughnessy — Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488] (August 25, 2026) · at 42:14 ▸ · at 1:15:04 ▸
When this applies
when a supplier has pricing power during a shortage and must decide whether to maximize short-term revenue or preserve a diversified buyer ecosystem
The takeaway
A dominant supplier facing a shortage protects long-term power by diversifying its customer base rather than just selling to whoever pays most.
Why it works
Letting the deepest pockets buy all scarce supply concentrates downstream power in a few customers, who can then extract more leverage over the supplier later; allocating strategically (and maintaining many mid-size buyers who need NVIDIA) preserves the supplier's long-term negotiating position.
One level up
Short-term profit-maximization under scarcity can undermine long-term structural power; deliberate underpricing/rationing can be a power-preservation strategy.
Where it stops holding
This stops working when the supplier has no real risk of customer concentration (e.g., truly commoditized product) or when competitors can supply the same good just as well, removing the leverage to ration.
Detail
situation: Immense demand for Blackwell chips far exceeds supply; NVIDIA could crank prices to clear the market.
decision: NVIDIA instead allocates strategically, favoring relationships and avoiding letting any single deep-pocketed buyer accrue too much compute/power; Jensen Huang also cultivates multiple NeoCloud providers like CoreWeave rather than selling tokens directly.
outcome: NVIDIA maintains a diverse, competitive ecosystem of buyers all dependent on and creating demand for its chips, with no single buyer able to threaten its position.
lesson: Suppliers with market power can trade short-term revenue for long-term structural control by deliberately fragmenting their customer base.
In their words
He wants to create a diverse community of NeoClouds and inference providers who are all jockeying to create demand for NVIDIA, such that if any one of them decides to, I don't know, vertical integrate or go with AMD or any other option, he's got three more people hungry to fill that position.