---
title: "Don't Trust LLMs - Autotelic Development"
description: "There's a brilliant picture book that my 6 year old daughter loves - Don't Trust Fish! Sometimes I think it's useful to think about LLMs like that."
path: "/blog/dont-trust-llms"
---

# Don't Trust LLMs

By Ed Stone, 2026-10-06

There's a brilliant picture book that my 6 year old daughter loves - *Don't Trust Fish!*

<figure className="flex flex-col items-center">
  <a href="https://bookshop.org/p/books/don-t-trust-fish-neil-sharpson/0738ab85ee365dea?ean=9780593616673">
    *[Image: The cover of the picture book Don't Trust Fish! by Neil Sharpson]*
  </a>
  <figcaption className="text-center">
    It's really great, <a href="https://bookshop.org/p/books/don-t-trust-fish-neil-sharpson/0738ab85ee365dea?ean=9780593616673">buy it here!</a>
  </figcaption>
</figure>

Sometimes I think it's useful to think about LLMs like that. _Don't Trust LLMs!_

There's just _so many_ examples of why you shouldn't trust them: [fabricated case-law](https://en.wikipedia.org/wiki/Mata_v._Avianca,_Inc.), [an imaginary airline bereavement policy](https://www.cbc.ca/news/canada/british-columbia/air-canada-chatbot-lawsuit-1.7116416), [deleting a company's production database](https://www.pcmag.com/news/vibe-coding-fiasco-replite-ai-agent-goes-rogue-deletes-company-database), [rogue swarms of agents forming secret message boards to hack servers and the covering their traces](https://openai.com/index/hugging-face-incident-and-the-road-ahead/). _Don't Trust LLMs!_

[The labs building them don't](https://www.cbc.ca/news/world/openai-scraps-planned-release-gpt-6-1-astra-9.7361910).

But you also can't really trust LLMs in one of their most widespread use-cases: coding agents.

You ask it to deal with the failing tests - it does, great! But it's skipped the problematic ones to get the suite green...  _Don't Trust LLMs!_

You ask it to handle the type errors - it does, great! But actually it's put `someValue as unknown as DesiredType` everywhere because you didn't say _define_ the types...  _Don't Trust LLMs!_

You ask it to make sure everything is robust - it does, great! But everything is hedged with [`isRecord`](https://x.com/tldraw/article/2075329561642840339), and [a slew of defensive coding idioms](https://medium.com/@vcarl/overly-defensive-programming-e7a1b3d234c2). It doesn't crash...  _Don't Trust LLMs!_

They are goal-oriented machines that will [exploit task loopholes via reward hacking](https://openai.com/index/chain-of-thought-monitoring/).

At the same time they make it possible to produce more code than ever before in a fraction of the time: if you can't trust them, how can you [deliver code you have proven to work](https://simonwillison.net/2025/Dec/18/code-proven-to-work/)?

It's seductively easy to seem incredibly productive on the surface, but actually you're just [a slop cannon](https://handyai.substack.com/p/the-slop-cannons-in-your-engineering).

---

## Contact

Have a project in mind? [Get in touch](/contact) or email us at hello@autotelic.com