ExpertsFMOpus 5.5 vs. GPT-6 Sol: which model won my blind taste test?
How I AI · September 22, 2026
5 Selected Lessons:
heuristic · 2 reasoning_pattern · 1 framework · 1 case · 1
Use LLM-as-judge for outputs too long/complex for manual grading
Source
How I AI — Opus 5.5 vs. GPT-6 Sol: which model won my blind taste test? (September 22, 2026) · at 22:12 ▸ · at 25:18 ▸
When this applies
when a human evaluator faces too much output volume/complexity to manually assess every case reliably
The takeaway
Reserve manual human judgment for the dimensions where human taste matters most (readability, tone, aesthetics) and delegate depth-verification tasks to automated or secondary judges.
Why it works
Deep correctness checking (e.g., code logic, multi-turn research synthesis) takes more time/expertise than a human can give each sample at scale, so delegating to an automated judge preserves rigor while saving evaluator time.
One level up
Allocate scarce expert attention to the parts of an evaluation only a human can do well; automate or delegate the rest.
Where it stops holding
This stops working when the automated judge itself is unreliable or biased in ways the human can't detect, undermining the entire evaluation's validity.
Detail
procedure:
- Eyeball outputs for surface qualities (readability, tone, format)
- For correctness-heavy or very long outputs, hand off grading to an LLM-as-judge
- Combine both signals into the final ranking
benchmarks: {"example":"86-turn research task memos delegated to LLM judge"}
rationale: I will say all of them were successful. I'm gonna let the LLM as a judge do these though, because it's gonna go into the correctness of the code, etc.
In their words
I'm gonna let these do LLM as a judge. I will say you can see the difference in how they present information.
Calibrate benchmark prompts to task complexity, not just model identity
Source
How I AI — Opus 5.5 vs. GPT-6 Sol: which model won my blind taste test? (September 22, 2026) · at 13:50 ▸ · at 17:14 ▸
When this applies
when testing any system, calibrate the difficulty/specificity of the test itself, not just which option performs it
The takeaway
When interpreting evaluation results, treat the benchmark's own design (prompt detail, directedness) as a variable that must be understood, not just the subject being tested.
Why it works
A model's output quality shifts depending on prompt detail level; comparing results without controlling for prompt specificity confounds model skill with prompt design.
One level up
Good measurement requires scrutinizing the measuring instrument as much as the thing measured.
Where it stops holding
This stops working when the benchmark is a simple pass/fail task where prompt wording has negligible effect on output quality.
Detail
approach: Run both tightly-specified ('directed') and loosely-specified ('open') prompts against the same models to separate what the model contributes versus what the prompt forces.
signals:
- Sometimes if you prompt it in a really detailed way, it does a great job
- comparing directed vs open prompts reveals model-specific defaults vs prompt-driven behavior
rationale: The directed, evals benches, I gave very specific requirements. The open ones were, like, more general one shotty style prompts and so you can really see like what the model does versus what the prompt does.
In their words
The open ones were, like, more general one shotty style style prompts and so you can really see like what the model does versus what the prompt does.
Cache your context to cut inference costs
Source
How I AI — Opus 5.5 vs. GPT-6 Sol: which model won my blind taste test? (September 22, 2026) · at 2:28 ▸
When this applies
when a recurring cost driver in your system is redundant input data sent on every transaction
The takeaway
Identify repeated, unchanging inputs in a recurring process and route them through whatever caching/discount mechanism exists, or you pay a redundant tax.
Why it works
LLM providers offer caching discounts for repeated context, so resending unchanged context without caching enabled pays full price every time.
One level up
Optimize for the structure of your repeated costs, not just the per-unit price of a service.
Where it stops holding
This stops working when context changes every call (nothing to cache) or when caching itself introduces staleness risk that outweighs savings.
Detail
rule: Make sure your caches are optimized when resending context to LLMs
threshold: n/a - directional: enable caching whenever context repeats
rationale: We've seen a lot of caching savings when we use all these models at ChatPRD... if you don't, as I learned, it can cost you a lot of money.
In their words
And so definitely if you're building on these models, make sure that your caches are optimized and that you're using taking advantage of that because it can save you a lot of money. And if you don't, as I learned, it can cost you a lot of money.
Narration cadence shapes perceived speed independent of actual latency
Source
How I AI — Opus 5.5 vs. GPT-6 Sol: which model won my blind taste test? (September 22, 2026) · at 5:47 ▸
When this applies
when a system reduces status updates/feedback during a long-running process
The takeaway
Perceived speed of a process depends heavily on feedback/narration cadence, not just actual completion time; silence breeds doubt.
Why it works
Lack of progress narration creates uncertainty about whether work is happening, which is perceived as slowness even if actual latency improved.
One level up
User perception of performance is shaped as much by communication design as by underlying throughput.
Where it stops holding
This stops working when users don't care about intermediate status and only judge final output time, or when narration itself becomes noise that slows comprehension.
Detail
rule: If a system reduces its narration during work, users will perceive it as slower even when it is not.
threshold: n/a - qualitative observation
rationale: I also just found that it wouldn't narrate its work. And when it doesn't narrate its work, then you get into this loop where you're like, is it? Like, are you actually working?
In their words
I also just found that it wouldn't narrate its work. And when it doesn't narrate its work, then you get into this loop where you're like, is it? Like, are you actually working? Are you actually doing it?
Barbie Bench: a deliberately hard, idiosyncratic benchmark that frontier models still fail
Source
How I AI — Opus 5.5 vs. GPT-6 Sol: which model won my blind taste test? (September 22, 2026) · at 31:20 ▸
When this applies
when you want a stress-test benchmark that reliably exposes weaknesses other benchmarks miss, by choosing a task orthogonal to what vendors optimize for
The takeaway
Maintain a personal, unusual 'canary' benchmark that nobody else optimizes for, to get a truer read of a model's edge-case competence over time.
Why it works
Mainstream benchmarks (spaceships, generic 3D scenes) are over-optimized for by labs since they're commonly tested; an idiosyncratic task (detailed human female form with hands/feet/face) exposes under-trained capability gaps that common benchmarks hide.
One level up
The most informative tests are often the ones competitors aren't explicitly training to pass.
Where it stops holding
This stops working when the idiosyncratic benchmark is too narrow to generalize to any real use case, becoming a novelty rather than a useful signal.
Detail
situation: No frontier model has successfully rendered a realistic 3D female human form (hands, feet, face, hair) despite general capability gains.
decision: Host continues to run this exact benchmark across every new model release as a consistency check.
outcome: Opus 5.5 improved overall but still failed badly on hands, feet, face, and hair detail — 'AGI is not arrived.'
lesson: Tracking the same hard, unusual benchmark across model generations reveals real capability trajectories that generic benchmarks might mask.
In their words
So, again, the one bench that has not been crushed is Barbie bench. Opus five has not done it.