AI agents are easy to use and hard to reason about.
For product and design leaders, it’s a risky combination. When something ends up going wrong (it will), users will not be blaming the model. They will blame your product.
I have been reading a lot about the Claude Code architecture, OpenAI agent documentation, and building some myself to understand how these systems behave. Along the way, I started mapping out a simple mental model for how agents think, loop, and execute.
This post’s goal isn’t to fully break-down everything on an implementation level. It’s more about giving non-engineering leaders some background info to evaluate agent behavior. It should help you spot weak assumptions and avoid shipping systems your team may not fully understand.
LLMs vs AI Agents
As a product leader deep in the software world, you need to understand the difference between these two technologies. They are NOT the same. When your engineering department comes to you talking about an LLM and an Agent, they’re talking about two different things.
The core technology with any AI system are LLMs or Large Language Models. Simply put, they take inputs (from users) and produce outputs (funnily enough, they actually output numbers aka tokens that end up getting translated into human readable text).
With this, you can talk to the LLM. This is how chat tools like ChatGPT work. They’re dead simple in this way. You talk, the LLM responds.
This can be useful in many contexts; getting direct answers to questions, finding information on the web, etc. However AI Agents take this to the next level.
The Agent Loop
Core to the architecture of an Agent is the Agent Loop. The LLM runs more than once. “Why do I even need to know this” you might be asking yourself. Well, you want to help your team perform, right? You have to understand what an Agent “Loop” entails.
Let’s start off with something you might have more familiarity with (hopefully). The human brain.
When you’re doing any sort of task, what’s really happening in that thick skull of yours? Our brains are very complex. Millions of neurons firing at any given moment. There’s loads of filtered context being pulled in at any given moment, where you are, who you are, why you’re doing things in the first place, checks and balances to tell you to stop when it makes sense to.
Agents are the same.
We’re trying to do the exact same thing with an Agent Loop. Applying checks and balances, pulling in context, consuming documents to understand its surroundings, etc. In order to do all of this, it needs to be able to have the ability to pull that context in. Prompting the task-giver for questions, looking for documents online, trying to understand the current codebase’s structure, applying its own checks and balances against the outputs it’s producing.
Ultimately, just as we’ve figured out how to fly based on birds’ bone structure, we’re doing the same thing in the case of AI. Agents are one step closer than an LLM to a human.
Here’s a diagram I whipped up to help contextualize what an Agent Loop consists of:
This specific Agent loop is more closely related to Claude Code’s structure, but I think we can apply it to a lot of other areas, not just programming.
Here are a couple of core phases that I was able to pull out:
-
The Reasoning Phase - The initial pass of an LLM to understand the current high-level context, the user’s ask, and rationalizing what needs to happen next.
-
The Exploration Phase - The phase after initial reasoning that allows the agent to build more context, ask questions, and put together an executive plan.
-
The Execution Phase - The final phase that allows the agent to build on top of itself, adding context, and attempting to solve problems.
These all come into play as we look at an agent from a higher level. This diagram below shows all of the different levers that AI agents can take advantage of:
There are three core spaces that we can think about here:
-
The Core LLM - The model has its own set of core capabilities that allow it to effectively navigate the world. Compute, context limits, training data. Most of the time, you won’t touch any of this (unless you have billions of $$$ to spend). You just need to understand it.
-
Agent Context - What Agents can “see”. Super important to manage this effectively. A badly managed context is a badly managed agent. Our brains consume relevant information depending on the situations. Agents need to work the same.
-
Agent Capabilities - These are the tools that the agent is either provided for by default, or extended on top of through developer written MCP (model context protocol; a bit like an API but AI specific) tools, executive access to resources, and other means of execution like API’s, uploaded documents, etc.
Take a step back, take a deep breath in, and take another look at those diagrams. How does everything connect?
All of these spaces have to be considered in order to create effective AI agents. Not knowing that an AI’s context window can handle only a few 1000 tokens can severely limit you on how to leverage context effectively.
Not knowing these fundamentals will ultimately be a foot-gun when you want to make improvements, or just simply build an AI Agent that actually works. That way you can create solutions that aren’t dragging the agent down, but building it up.
perhaps controversial, I watched about half of that interview with Musk and was totally wigged out. I’m unsure how peeps are excited / onboard with the idea that AI will take up so many jobs. Seems like we have the ability not to lean into that idea, as a society but what do I know. LOL