Don't Trust LLMs
Ed Stone
There's a brilliant picture book that my 6 year old daughter loves - Don't Trust Fish!
It's really great, buy it here!
Sometimes I think it's useful to think about LLMs like that. Don't Trust LLMs!
There's just so many examples of why you shouldn't trust them: fabricated case-law, an imaginary airline bereavement policy, deleting a company's production database, rogue swarms of agents forming secret message boards to hack servers and the covering their traces. Don't Trust LLMs!
But you also can't really trust LLMs in one of their most widespread use-cases: coding agents.
You ask it to deal with the failing tests - it does, great! But it's skipped the problematic ones to get the suite green... Don't Trust LLMs!
You ask it to handle the type errors - it does, great! But actually it's put someValue as unknown as DesiredType everywhere because you didn't say define the types... Don't Trust LLMs!
You ask it to make sure everything is robust - it does, great! But everything is hedged with isRecord, and a slew of defensive coding idioms. It doesn't crash... Don't Trust LLMs!
They are goal-oriented machines that will exploit task loopholes via reward hacking.
At the same time they make it possible to produce more code than ever before in a fraction of the time: if you can't trust them, how can you deliver code you have proven to work?
It's seductively easy to seem incredibly productive on the surface, but actually you're just a slop cannon.
Find out more about the work we've done, and the services we offer, or just contact us right now to discuss your project.