How LLM tool calling actually works

This is what comes back from the model when an agent decides to read a file:

{
  "stop_reason": "tool_use",
  "content": [
    { "type": "text", "text": "Let me look at how renewals are calculated." },
    {
      "type": "tool_use",
      "id": "toolu_01A7...",
      "name": "read_file",
      "input": { "path": "app/models/subscription.rb" }
    }
  ]
}

That’s an Anthropic API response, trimmed. Nothing has been read. No file was opened on any server anywhere. The model produced some text, and then it produced a block that says “I would like read_file to be called with this path”, and then it stopped, and the stop_reason tells you why it stopped. Whether the file gets read is entirely up to the code that made the request.

I described this loop briefly when I wrote about the agent that files my expenses last year, and I’ve used it every day this year through Claude Code, so I wanted to go through it properly, because I keep meeting people (some of them engineers) who think the model is somehow reaching out and doing things. It isn’t. Every agent I know of, from a 250-line Python script to Claude Code, is a program that sends text to a model, gets text back, and decides what to do with it.

Telling the model what it can call

You describe each tool in the request, alongside the messages:

{
  "name": "read_file",
  "description": "Read a UTF-8 text file from the repository. Paths are relative to the repo root. Returns at most 2,000 lines.",
  "input_schema": {
    "type": "object",
    "properties": {
      "path": { "type": "string", "description": "e.g. app/models/user.rb" }
    },
    "required": ["path"]
  }
}

A name, a description in plain English, and a JSON Schema for the arguments. OpenAI’s version has a slightly different shape (a function with parameters) and the same three parts.

What happens to that on the provider’s side isn’t fully public, but the general idea is. The tool definitions are turned into text and put into the model’s context, usually near the start, in a format the model was trained on. The model has been trained, on a lot of examples, to respond to a request it can’t answer from what it already has by writing a call in a particular structured format, and to stop after writing it. The API spots that format in the output, parses it, and hands it back to you as a tool_use block instead of text.

So the description is part of the prompt, which has two consequences I’ve learned the hard way. The first is that the description is the most important thing you write. “Read a file” gets you a model that tries to read directories with it. “Returns at most 2,000 lines” gets you a model that knows to ask for a range on a big file. Most of my improvements to my own little agents this year have been edits to tool descriptions, not to the main prompt.

The second is that tool definitions cost tokens on every request. One definition like the one above is a hundred-odd tokens. A setup with forty tools, each with a careful description and a few parameters, can easily be ten thousand tokens before the conversation has even started, and the model has to read all of it, every time, and pick the right one from forty. That’s the main reason I’ve stayed away from connecting lots of MCP servers at once. More on MCP below.

Your code runs it

Back to the response at the top. Your code sees stop_reason: "tool_use", finds the tool_use block, and runs whatever read_file means in your program. Then it sends the model a new request, with the whole conversation so far plus two more messages: the assistant’s response exactly as it came back, and a user message containing the result:

{
  "role": "user",
  "content": [
    {
      "type": "tool_result",
      "tool_use_id": "toolu_01A7...",
      "content": "class Subscription < ApplicationRecord\n  ..."
    }
  ]
}

The tool_use_id matches the result to the request. The model reads the file contents as part of its input on the next turn, and then either answers or asks for another tool. Round and round, until it stops with end_turn instead of tool_use, or until your code decides it’s had enough.

That box on the left saying “allowed?” is where all the safety in an agent lives. When Claude Code asks me “Allow this command?”, that’s the harness pausing between the tool_use block arriving and running the command. The model has already asked. The question is whether my program does it. Claude Code’s own Read, Edit and Bash are tools in exactly this sense, with definitions sent in every request, and so is everything else it can do.

A few details that matter

The model can ask for several tools at once. If it wants three files, it can put three tool_use blocks in one response, and you send back three tool_result blocks in one message, each with its own id. Agents feel a lot faster when the harness runs those in parallel instead of one after another.

Errors are results too. If the file doesn’t exist, you don’t throw an exception and end the conversation. You send back a tool_result with the error message and is_error: true, and the model will usually read it and try something else, like listing the directory. A lot of what makes an agent seem clever is just that it gets to see its own mistakes and has another go.

You can steer whether it calls a tool. tool_choice lets you leave it to the model, force it to use some tool, force a specific one, or forbid tools for this turn. Forcing a specific tool is a cheap way to get structured output out of a model, and was the standard trick for that for a while: define a tool whose input schema is the shape you want, force the model to call it, and never actually run anything.

The arguments might not match your schema. The model is generating JSON one token at a time, and on most setups nothing stops it from writing an argument your schema doesn’t allow, or leaving out a required one, or calling a tool that doesn’t exist. It’s rare with current models, and it isn’t zero, so validate before you run anything. Both big providers now offer a strict mode that constrains generation so the output has to match the schema: OpenAI since last year, and Anthropic in beta since last month. That guarantees the shape. It guarantees nothing about whether the path the model asked for is a sensible one to read.

The context

The thing that surprised me most when I first wrote one of these is that the API remembers nothing. Every request contains the entire conversation: the system prompt, every tool definition, every message, every tool call and every result, from the start. The model reads all of it again each turn.

So the cost of a session grows faster than its length. Take a small agent session with a system prompt and tool definitions of 3,000 tokens, where each tool call and its result add about 1,500 tokens to the conversation. After 20 tool calls, the last request is 3,000 + 20 × 1,500 = 33,000 tokens. That’s fine. But you sent 21 requests to get there, each one bigger than the last, and the total input across all of them is

21 × 3,000 + 1,500 × (0 + 1 + … + 20) = 63,000 + 1,500 × 210 = 378,000 tokens.

That’s more than eleven times the size of the final conversation, and it’s quadratic: double the number of tool calls and the total roughly quadruples.

At Claude Sonnet 4.5’s list price of $3 per million input tokens, those 378,000 tokens are about $1.13, for one short session, before any output. Prompt caching is what makes agents affordable. The provider keeps the processed prefix of your request for a few minutes, and when the next request starts with the same prefix, which in an agent loop it always does, you pay a fraction of the price for that part. With Anthropic’s cache that’s a tenth of the normal input price for reads, and a quarter extra for writing the new part. Worked through for the same session, it’s about 345,000 tokens read from cache (about 10 cents) and 33,000 written to it (about 12 cents), or roughly 23 cents instead of $1.13. It’s also why the order of things in the request matters. Anything that changes near the start, like a timestamp in the system prompt, breaks the cache for everything after it.

And context windows are finite. On a long session, the conversation eventually doesn’t fit, and the harness has to do something about it: summarize older turns, drop old tool results, or start over with notes. Claude Code’s /compact is the first of those. The sub-agents I wrote about in September are mostly a way of not filling the main context in the first place. A sub-agent’s thirty file reads happen in its own conversation, and only its summary comes back as a tool result in yours.

Where MCP fits

The Model Context Protocol, which Anthropic published a year ago and which everyone seems to support now, often gets described as a way for models to use tools. That’s true from a distance and misleading up close. The model knows nothing about MCP.

An MCP server is a program that can tell a client “here are my tools, with names, descriptions and schemas” and “here’s the result of running this one with these arguments”. The client, say Claude Code, connects to the server, gets the list, and converts each one into a tool definition in the requests it sends to the model. When the model writes a tool_use block for one of them, the client sends it to the right server and puts the server’s answer into a tool_result. From the model’s point of view an MCP tool and a built-in one are indistinguishable. MCP standardizes the left-hand side of my diagram, which is useful, because it means one server works with every client. It doesn’t change anything on the right.

It does make it very easy to add tools, which is where the token cost from earlier comes back. Each server you connect adds its definitions to every single request. I tried a handful on my own machine last winter, and the one I removed first was a server with about thirty tools that I used two of.

What follows from all this

Once you’ve seen the loop, a lot of agent behavior stops being mysterious. The agent “forgot” what you said an hour ago because it was summarized away to make room. It “decided” to delete a file because a tool that can delete files was in its list, and the harness let it. And a web page or a support ticket can steer it, because a tool result is just more text in the same conversation, and the model reads text from a tool result the same way it reads text from you.* That’s the reason I keep saying the permissions matter more than the prompt. The model can be talked into asking for anything. What it gets is decided by a few hundred lines of ordinary code on the left of the diagram, and that’s code you can read, test and review like any other.

* This is prompt injection, and it’s the one problem with this design that nobody has a general fix for yet. Providers train models to be suspicious of instructions inside tool results, which helps. The only defense I fully trust is the one from my expenses agent: don’t give the loop a tool that could do real damage if the model were persuaded to use it.