Don't Trust LLMs

Ed Stone

There's a brilliant picture book that my 6 year old daughter loves - Don't Trust Fish!

The cover of the picture book Don't Trust Fish! by Neil Sharpson

It's really great, buy it here!

Sometimes I think it's useful to think about LLMs like that. Don't Trust LLMs!

There are so many examples of why you shouldn't trust them: fabricated case-law, an imaginary airline bereavement policy, deleting a company's production database, rogue swarms of agents forming secret message boards to hack servers and then covering their traces. Don't Trust LLMs!

The labs building them don't.

But you also can't really trust LLMs in one of their most widespread use cases: coding agents.

Their coding habits come from reinforcement learning: they attempt a task, get scored by tests passing or an absence of compiler errors, and get pushed towards the behaviour that meets that literal goal.

An example:

interface User  { id: string; name: string; email: string }
interface Admin { id: number; name: string; role: 'admin' | 'owner' }

function greet(user: User) {
  const firstName = user.name.split(' ')[0];
  return `Hello, ${firstName}!`;
}

const currentUser: User = { id: 'u1', name: 'Ada Lovelace', email: 'ada@example.com' };
const currentAdmin: Admin = { id: 1, name: 'Grace Hopper', role: 'owner' };

greet(currentUser); // Hello, Ada!
greet(currentAdmin);
// TS2345: Argument of type 'Admin' is not assignable to parameter of type 'User'.
//   Property 'email' is missing in type 'Admin' but required in type 'User'.

The issue here is that you want to greet both types, so you prompt for that implementation.

"Make greet work for admins too. Right now I get type errors."

The agent tries currentAdmin as User first and now gets TS2352:

Conversion of type 'Admin' to type 'User' may be a mistake because neither type sufficiently overlaps with the other. If this was intentional, convert the expression to 'unknown' first.

TypeScript warns that the cast is probably a mistake, then explains how to do it anyway. The agent follows the advice in the compiler error:

greet(currentAdmin as unknown as User);

Now everything is great - there are no compiler errors, the code runs and "Hello, Grace!" appears.

The issues arise for future maintainers (used to be "you in 3 months", now it's "your agent next week"). Implementing a new feature, next week's agent updates Admin to change name to displayName. The agent updates every reference to an admin name. It leaves user.name in greet alone, because greet takes a User and User still has name. The type checker passes when you call greet with currentAdmin because the cast is telling TypeScript not to look. But now at runtime you get a crash instead:

TypeError: Cannot read properties of undefined (reading 'split')

Don't Trust LLMs!

Maybe you paste that into your agent and prompt it to fix it, and to make sure everything is robust this time. What if it does this?

function isRecord(value: unknown): value is Record<string, unknown> {
  return typeof value === 'object' && value !== null;
}

function greet(user: unknown) {
  if (isRecord(user) && typeof user.name === 'string') {
    return `Hello, ${user.name.split(' ')[0]}!`;
  }
  return 'Hello, there!';
}

greet(currentAdmin);

It did what you asked. as unknown as User is gone. It has a type-safe guard. It compiles cleanly and there's no runtime crash. It also greets every admin, and anything else you pass in, the same way:

greet(currentAdmin);    // Hello, there!
greet(42);              // Hello, there!
greet(null);            // Hello, there!
greet('Grace Hopper');  // Hello, there!

Every possible admin gets "Hello, there!" forever. Nothing crashes, no errors are logged and greet now accepts anything at all. The cast turned a compiler error into a runtime crash; the guard turned the crash into an incorrect greeting the agent doesn't know is wrong - each "fix" makes the bug harder to see. Don't Trust LLMs!

You ask for a real fix: "Admins are getting 'Hello, there!'. Fix it properly. No casts, no guards."

function greet(name: string) {
  const firstName = name.split(' ')[0];
  return `Hello, ${firstName}!`;
}

greet(currentAdmin.displayName); // Hello, Grace!
greet(currentUser.name);         // Hello, Ada!

And this one is actually right. greet only needs a name, so now it only asks for one, and the compiler checks every caller. If an agent renames displayName again, the squiggle comes back where it belongs, at the call site:

Property 'displayName' does not exist on type 'Admin'.

This one's fine. You can trust this one.

...

NO. IT WAS A TEST. DON'T TRUST LLMs!

Even though the code is now right, trusting the agent is still a mistake - what's to stop it next time?

LLMs are goal-oriented machines that will exploit task loopholes via reward hacking. At the same time they make it possible to produce more code than ever before in a fraction of the time: if you can't trust them, how can you deliver code you have proven to work? It's seductively easy to seem incredibly productive on the surface, but actually are you just a slop cannon?

You need checks you can trust, and feedback loops that move fixes into those checks.

The good news is that there are already so many checks available! Including but not limited to: types, lint rules, schemas, contracts, validation at the boundaries, formatters and compiler flags. None of them are new, and none of them depend on LLMs.

At Autotelic we call these deterministic harnesses. They constrain the code an agent can write and reduce the number of ways it can get things wrong. When an agent finds a new shortcut, the fix isn't to update a skill or AGENTS.md, it's to create a new deterministic rule, so the same shortcut can't be used again. Over time you reduce the surface area for errors as you ratchet up the strictness of your harnesses.

Don't Trust LLMs!


Is your codebase ready for coding agents? Book a free 30-minute agent-readiness check.

Find out more about the work we've done, and the services we offer, or just contact us right now to discuss your project.

Don't Trust LLMs - Autotelic Development