ExpertsFMUnderwriting Superintelligence: Backing Agents you can Sue — Rune Kvist, AIUC
Latent Space: The AI Engineer Podcast · September 16, 2026
5 Selected Lessons:
heuristic · 2 reasoning_pattern · 2 case · 1
Independent third-party broker needed wherever trust gaps exist between parties with asymmetric incentives
Source
Latent Space: The AI Engineer Podcast — Underwriting Superintelligence: Backing Agents you can Sue — Rune Kvist, AIUC (September 16, 2026) · at 25:27 ▸ · at 28:40 ▸ · at 1:01:38 ▸
When this applies
whenever one party must rely on information from another party that has an incentive to shade the truth
The takeaway
No industry allows the regulated party to audit itself; wherever this pattern of informed-but-incentivized vs trusting-but-uninformed exists, a neutral technical broker is structurally necessary.
Why it works
A party with deep domain expertise but misaligned incentives (wanting to keep deploying/releasing) cannot be fully trusted to self-report risk; a neutral party with technical rigor but no stake in the outcome can bridge the credibility gap, similar to financial auditors.
One level up
Trust infrastructure scales by inserting independent verification at the exact point where interests and information diverge.
Where it stops holding
This stops working when no credible neutral party can be found that both parties trust, or when the technical complexity is too specialized for any third party to independently verify.
Detail
approach: When government distrust of self-reported lab risk-data threatens adoption, insert a broker that: (1) specifies one public, inspectable standard, (2) defines the exact threat models and required guardrails/tests per risk, (3) runs technical audits with transparent, repeatable methodology.
signals:
- a party with expertise but incentive to shade truth
- a party with authority but lacking expertise
- no existing neutral verifier
rationale: There's no other industry where you allow people to audit themselves; a broker that both runs frontier-grade technical evals and speaks the legible, trusted language of an audit report is the natural solution.
In their words
There's no other industry where you allow people to audit themselves. So there's going to be a need for a third party that can take the rigor of the labs to run frontier technical evals, but can also speak legible trusts in the way that the government trusts PwC to go and run financial audits.
Independent watchdogs survive even hyper-concentration because self-auditing is structurally impossible
Source
Latent Space: The AI Engineer Podcast — Underwriting Superintelligence: Backing Agents you can Sue — Rune Kvist, AIUC (September 16, 2026) · at 1:21:18 ▸
When this applies
when assessing whether an independent oversight/verification business has durable value even if the industry it watches becomes extremely concentrated
The takeaway
The need for independent verification is driven by structural incentive misalignment (race dynamics), not by lack of competence or good intent — so it persists even as the industry concentrates into a few very capable players.
Why it works
Even perfectly well-intentioned, highly capable incumbents face race dynamics and incentive pressures (competitive pressure to cut corners, incentive to withhold unfavorable information) that make self-certification structurally untrustworthy regardless of how smart or caring the people inside are.
One level up
Oversight businesses are durable precisely because the thing they check for (self-interest distorting self-report) is a structural feature of any competitive race, not a fixable bug.
Where it stops holding
This stops working, by Rune's own admission, if the technology triggers a sovereignty-level event (e.g., full nationalization), where 'every company is the government,' a scenario his business isn't built to operate in.
Detail
approach: Project forward: ask whether the core incentive problem (labs can't be their own watchdog in a race) still holds under the hypothetical future state; if yes, oversight remains necessary.
signals:
- labs still compete against each other (race dynamics persist)
- labs retain incentive to withhold unfavorable info from government
- no sovereignty-level nationalization event has occurred
rationale: There's one job that the labs can never do for themselves, which is to be their own watchdog... The problem is not whether they care. The problem is that they will all be stuck in a race where they might have incentives to cut corners.
In their words
So I think kind of to extent that you believe this frame of like you'll see hyper concentration, like the labs will kill all the startups
Capability is not the binding constraint on adoption — trust is
Source
Latent Space: The AI Engineer Podcast — Underwriting Superintelligence: Backing Agents you can Sue — Rune Kvist, AIUC (September 16, 2026) · at 8:00 ▸
When this applies
when a product/technology is demonstrably better than the status quo yet adoption stalls
The takeaway
Once a technology crosses a 'good enough' capability threshold, the adoption bottleneck shifts from capability to the ability to make credible promises about what it will and won't do.
Why it works
Institutions (governments, banks, hospitals) need a way to make a go/no-go decision; without a mechanism to make promises or quantify risk, they default to no, regardless of actual capability.
One level up
As power/autonomy of any system increases, the risk surface grows faster than trust infrastructure, making trust (not raw capability) the scarce resource.
Where it stops holding
This stops working when the downside risk of failure is trivial or already well-understood/insurable, in which case capability remains the real bottleneck.
Detail
rule: If a system is highly capable but not adopted, check whether the blocker is capability or trust/liability infrastructure.
threshold: A technology can be objectively superior (e.g., safer-than-human driving) and still be blocked for years absent promise infrastructure.
rationale: More intelligent/autonomous AI is more valuable but also carries a growing risk surface that institutions can't yet quantify or insure against.
In their words
So in that particular instance, what's clear is that the binding constraint on AI being useful is not capability, but instead liability or risk or trust.
Stress-test for the adversary, not just the average case
Source
Latent Space: The AI Engineer Podcast — Underwriting Superintelligence: Backing Agents you can Sue — Rune Kvist, AIUC (September 16, 2026) · at 21:42 ▸
When this applies
when building any product where most engineering time goes to making the happy path work well
The takeaway
The gap between 'looks like it has the right architecture' and 'the architecture actually holds under adversarial pressure' is usually where real risk hides, and it requires a distinct expertise (red-teaming, literature review on evasion techniques) to close.
Why it works
Most teams naturally optimize for what customers see most often (the good/average case) because that's where visible value is created; adversarial/corner-case robustness requires deliberately imagining a hostile actor, which is a different skill and mindset most early teams haven't hired for.
One level up
Any safeguard that has not been tested against a motivated adversary should be assumed untested, regardless of how reasonable it looks on paper.
Where it stops holding
This stops working when the product has no adversarial surface (no one benefits from breaking it), making stress-testing low value.
Detail
rule: Before shipping guardrails, explicitly model the adversary: what tricks/framings would someone use to defeat this filter, and have you tested against them (e.g., reading academic literature on evasion)?
threshold: It 'often takes a while' before a startup hires its first security person — a signal the adversarial mindset is usually missing early.
rationale: Most companies have the right kind of guardrail architecture in place; the problem is that it's fiddly and untested against real adversarial corner cases.
In their words
The most important thing is that a lot of companies have not done a serious stress test. They spend most of the time, perhaps rightly so, optimizing for how does it work in the good case, the average case.
UL's for-profit/non-profit split as proof for-profit standards serve customers better
Source
Latent Space: The AI Engineer Podcast — Underwriting Superintelligence: Backing Agents you can Sue — Rune Kvist, AIUC (September 16, 2026) · at 38:18 ▸ · at 41:00 ▸
When this applies
when designing an institution meant to set industry-wide safety/quality standards and deciding its ownership/incentive structure
The takeaway
A standard-setting body should be structured so its revenue depends on the standard staying rigorous and useful to the people it serves, not just on maintaining the appearance of legitimacy.
Why it works
Nonprofits lack both the profit-driven race-to-the-bottom risk AND the customer-responsiveness that a paying customer relationship forces; for-profits with the right incentive alignment (e.g., tied to insurers) stay responsive without hollowing out quality.
One level up
Design institutional incentives so that the entity's survival requires it to keep being truthful and useful, rather than relying on good intentions alone.
Where it stops holding
This stops working when the for-profit entity's revenue incentives diverge from accuracy (e.g., if paid directly by the audited company with no independent check), recreating a race to the bottom.
Detail
situation: Cybersecurity standards today are mostly run by nonprofits; UL was originally founded by insurers worried about houses burning down.
decision: UL recognized a for-profit entity better served customers and spun one out alongside the nonprofit; AIUC named itself after UL and chose to be for-profit, working closely with insurers.
outcome: AIUC believes this alignment (for-profit + insurer partnership) creates the feedback loop needed to stay both responsive to customers and rigorous.
lesson: For-profit structure, when tied to a party with skin in the game (insurers), can produce better standards than well-meaning nonprofits lacking customer feedback loops.
In their words
Today, UL has a for-profit entity and a non-profit entity what they've recognized, they spun out, they started a non-profit, they spun out a for-profit because what they recognized was like, hey, actually, to serve customers well, you need a for-profit entity.