Tool Use and Function Calling in AI Agents

Tool use and function calling are the mechanism that turns a language model from a very expensive autocomplete into something that can actually do things. That's the whole story. Everything else, the protocols, the benchmarks, the provider wars, is just context around that one shift. So let's make sure the foundation is solid before we get into the weeds.

Here's the baseline problem: LLMs are trained on static data. They know what they learned, and that's it. No live weather. No current stock prices. No ability to write a file, query a database, or send a message. Without some external mechanism, a language model asked "what's the weather in Tokyo?" will either guess, refuse, or confidently make something up. None of those are good.

Function calling is that external mechanism. The model outputs a structured JSON object that says, in effect, "call this function with these arguments." The host application, meaning the code you wrote, actually runs the function. The result comes back. The model reads it and continues. That's the loop.

The key thing to internalize here: the model never touches the outside world directly. It fills out a work order. Execution happens elsewhere. This isn't a limitation, it's the design. It keeps the model in its lane (reasoning and language) while the application handles the actual operations.

A quick note on terminology before we go further: OpenAI introduced "function calling" in June 2023 (it's now under the tools parameter). Anthropic calls it "tool use." Google Gemini calls it "function calling." The open-source community uses "tool calling" pretty interchangeably. The naming is all over the place. The underlying mechanism is identical across all of them.

The Sequence That Makes a Tool Call Work

There are five steps. They're not complicated, but every one of them matters.

Step 1: The tool catalog. You, the developer, describe the available functions in JSON Schema. Each entry gets a name, a description, and typed parameters with required fields. This catalog gets passed to the model alongside the user's prompt.

Step 2: The model decides. The model reads the prompt and the catalog together. It decides whether a tool call is needed at all. Sometimes it just answers directly. When it determines a tool is warranted, it moves to step three.

Step 3: Structured emission. Instead of writing prose, the model outputs a tool call object. Function name. Populated arguments. That's it. No narrative, no explanation, just the structured call.

Step 4: Application execution. Your application picks up that call and runs the actual function. API request, database query, file operation, whatever it is. The model is not involved in this step at all.

Step 5: Result incorporation. The result comes back as a new message in the conversation. The model reads it and either formulates a final response or decides it needs another tool call to get there.

That last point is important. The loop can repeat. A model can chain tool calls across multiple turns, building toward a result that required several external operations. Ask it to "find all open bugs assigned to me, summarize them, and create a Slack message draft" and you might be looking at three or four tool calls in sequence before you get an answer.

This is also why malformed output is such a real problem. An ambiguous or invalid JSON call breaks the chain at step four. Schema compliance isn't cosmetic. It's what keeps the whole thing running.

How OpenAI's Structured Outputs and Strict Mode Changed Reliability

When OpenAI launched function calling in June 2023, schema adherence was "best effort." The model tried to match the schema you provided. It didn't always succeed.

In production, this caused real headaches. Downstream code expecting a specific typed JSON object would break when the model went slightly off-script. Maybe it returned a string where you expected an integer. Maybe it added a field that wasn't in the schema. Either way, your code breaks, and debugging it is not fun.

In August 2024, OpenAI introduced strict: true on the gpt-4o-2024-08-06 model. It combines constrained decoding with model training to guarantee schema-valid outputs. Not "usually valid." Guaranteed.

The trade-off is real though. Strict mode doesn't support parallel tool calls in the same request. For multi-tool workflows where you want the model to invoke several tools simultaneously, that's a meaningful constraint. You're trading parallelism for reliability.

For agentic systems where tool calls chain across many steps, that trade-off often makes sense. A single malformed output in a five-step chain corrupts everything downstream. Reliability per call matters a lot more when calls are chained.

How Anthropic and Google Approach the Same Problem Differently

All three major providers support tool use. None of them do it exactly the same way, and the differences are worth understanding because they point to different strengths.

Anthropic uses a content-block architecture that keeps tool call outputs cleanly separated from text responses. Less ambiguity when you're parsing results. More importantly, Anthropic offers server-side tools: web search, a Python code execution sandbox, and a text editor that all run on Anthropic's own infrastructure. No other major provider does this. The practical effect is that entire categories of infrastructure work disappear from the developer's plate.

Google Gemini supports streaming argument construction, which cuts latency in real-time UIs. If you're building something where responsiveness is the priority, that matters. The friction point with Google is deployment. An agent built against AI Studio authentication often requires re-architecture to run on Vertex for enterprise production use. Teams that don't plan for this early tend to feel it later.

The reliability comparison is where this gets interesting. Per Rhumb AN Score data from Q1 2026, Anthropic leads tool-calling reliability at 8.4 out of 10. Google sits at 7.9. OpenAI comes in at 6.3.

That gap compounds fast in agentic workflows. If a tool call is 90% reliable and you make ten of them in sequence, your end-to-end success rate is roughly 35%. A reliability difference of one or two points per call multiplies across turns. At scale, you're not comparing individual calls anymore. You're comparing entire workflow outcomes.

The practical framing: Anthropic for infrastructure simplicity and raw reliability. Google for latency-sensitive real-time applications. OpenAI when strict schema compliance is the non-negotiable.

What Open-Source Models Can Now Do With Tool Calling

A couple of years ago, the honest answer to "can I run agents on my own infrastructure?" was "sort of, with a lot of caveats." That answer has changed.

Meta's Llama 3.1 and 3.2 models gained native tool calling support in 2024. They're production-ready via Ollama for standard use cases, and vLLM handles more demanding workloads. Mistral added tool calling through an OpenAI-compatible interface, which means teams already using OpenAI tooling can switch with minimal changes.

Qwen 3 and Qwen3 Coder are notable for something specific: long-horizon reasoning and recovery from execution failures. That last part is underrated. A model that can handle a failed tool call gracefully, figure out what went wrong, and try a different approach is categorically more useful than one that just stops or loops. That's the difference between a capable agentic model and a brittle one.

On the Berkeley Function-Calling Leaderboard V3 (June 2026), open-weight Chinese frontier models now lead the public rankings. Zhipu AI's GLM-4.5 sits at 77.8%, with Alibaba's Qwen3 variants clustered around 71 to 72%.

What the leaderboard ceiling also reveals: even the best models still struggle with multi-turn memory management, long-horizon reasoning, and knowing when not to invoke a tool. The single-call problem is largely solved. The hard stuff remains hard across all model types.

For teams that want to run agents on their own infrastructure, pick their own models, or avoid per-token API costs, the capability gap with proprietary models has narrowed meaningfully. The options are real.

MCP: Why a Universal Protocol for Tool Discovery Emerged

Here's the structural problem that existed before November 2024: every application that wanted to connect an LLM to external tools had to build its own integration logic. The same GitHub API wrapper would be reimplemented differently for every agent framework. Repeatedly. By many different teams. This is about as fun as it sounds.

Anthropic released the Model Context Protocol (MCP) in November 2024. The reported origin is kind of funny: developer frustration with copying code between Claude Desktop and an IDE. A mundane annoyance that turned out to point at a real structural gap in the ecosystem.

The right way to think about it: if function calling is the mechanism, MCP is the specification. It standardizes how tools are discovered, described, and invoked across providers and applications.

The "USB-C for AI applications" analogy gets used a lot, and it's apt. One connector. Any model communicates with any tool through one interface, rather than a different cable for every combination.

Architecturally, MCP separates the tool server (which exposes capabilities) from the client, meaning the model or agent, with a clean boundary between them. Either side can evolve independently. That matters for long-term ecosystem health.

In December 2025, MCP was donated to the Agentic AI Foundation under the Linux Foundation, the same governance structure that stewards Kubernetes and PyTorch. That's a clear signal of intent: vendor-neutral infrastructure, not an Anthropic-controlled standard.

How Fast MCP Spread and What the Adoption Numbers Actually Show

The growth trajectory here is unusual enough to be worth stating plainly.

MCP had roughly 100,000 SDK downloads in its launch month of November 2024. By March 2026, that number was 97 million monthly downloads. A 970x increase in 18 months.

For comparison: the React npm package took approximately three years to reach 100 million monthly downloads. MCP reached comparable scale in 16 months.

As of May 2026, the official MCP Registry counted 9,652 server records. The GitHub repository for MCP servers had over 86,000 stars and nearly 11,000 forks.

The cross-provider adoption happened fast. OpenAI adopted MCP across its Agents SDK and Responses API in March 2025. Google DeepMind confirmed Gemini support in April 2025. The protocol went from Anthropic-only to industry-wide in under six months.

For enterprise production use, Stacklok's 2026 software report puts 41% of surveyed organizations in limited or broad production with MCP servers. That's the most reliably sourced figure available.

The most concrete real-world outcome so far is Block (the company behind Square and Cash App). Block co-developed MCP with Anthropic and built Goose, an open-source MCP-compatible agent deployed to thousands of employees. The reported results: 50 to 75% time savings on common tasks, with work that previously took days completing in hours. The MCP servers connect to Snowflake, GitHub, Jira, Slack, Google Drive, and internal APIs.

That's not a demo. That's production infrastructure at a major financial technology company.

The Performance and Cost Math That Shapes How Agents Are Designed

This is the part that determines whether your agent is actually deployable or just impressive in a notebook.

Parallelism vs. sequential execution. Per the LLMCompiler paper published at ICML 2024, parallel tool calls reduce end-to-end latency by up to 3.7x compared to sequential execution. If your task allows it and your model supports it, parallelism is worth building for.

The token cost of tool definitions. Each tool definition adds roughly 100 to 300 input tokens to every request. A system with 20 tools adds somewhere between 2,000 and 6,000 tokens per request. At current GPT-4.1 pricing, that's approximately $0.01 to $0.06 per request. That sounds small.

At millions of requests per day, it is not small. Tool count is a direct cost lever.

A few practical mitigations that actually work:

• Remove unused tools from the catalog. Don't pass everything always.
• Consolidate related tools into one with an action enum parameter. Fewer definitions, simpler model decisions.
• Use Anthropic's prompt caching for stable tool catalogs. It cuts repeated tool definition costs by roughly 90%.
• When your catalog exceeds around 50 tools, use embedding similarity to dynamically retrieve only the relevant subset. The model cannot reliably reason across a very large catalog in a single context window.

These aren't theoretical optimizations. They're the difference between an agent that's economically viable at scale and one that quietly eats your infrastructure budget.

Where the Hard Problems in Tool Use Actually Sit Today

The state of tool use in mid-2026 is genuinely impressive and genuinely incomplete at the same time. Both things are true.

The Berkeley Function-Calling Leaderboard V4 puts the highest-scoring model at 77.47% overall accuracy. That means roughly one in four function calls fails to match expected behavior under benchmark conditions. And the benchmark doesn't capture everything: recovery from tool execution failures, cascading errors across long chains, graceful degradation when a tool is unavailable. Real conditions are harder than benchmark conditions.

The "when not to call" problem is real and underappreciated. Models trained to use tools aggressively will over-invoke. They'll trigger a tool call for a question the context window already answers. That adds latency and cost for no reason. Calibrating this is a genuine engineering challenge, not an edge case.

The enterprise authentication gap in MCP is worth naming specifically. When an AI agent connects to an MCP server for Slack or GitHub, the enterprise identity provider sees the user login. It does not see the agent connection. There's no native SSO support today. Full visibility and policy control over what the agent invoked, and why, are not currently built into the protocol.

Multi-turn memory and long-horizon reasoning remain the hardest open problems. The single-call case is largely solved. Maintaining coherent tool-use state across many turns, across a long-running agentic workflow, is not.

For organizations deploying agents at scale, these aren't abstract research problems. Reliability per call, visibility into agent actions, and control over when an agent is allowed to act are the variables that determine whether agent automation is actually trustworthy. Getting the JSON right was the beginning. Building systems that are auditable, controllable, and reliable across hundreds of chained decisions is the work that's still in progress.