AI building blocks: What is AI Agent Tool Calling

AI agent tool calling gives AI features the ability to take action, instead of simply generating text.

Up until recently, most new AI features I’ve seen have been some form of a chatbot or text generation feature. RAG makes these text generation features more knowledgable and relevant, but these LLM wrapper features are still disconnected from the rest of a user’s workflow and tech stack.

Tool calling gives AI agents the ability to perform tasks outside of the constraints of your product, whether it be searching the internet, getting weather data from external APIs, or updating data in your users’ third-party apps. If you’re building an AI product, you have to understand how tool calling works, which is exactly what we’ll cover today.

Tool calling basics

Let’s start with a few definitions:

Tool: An action (defined as a function in code) that an agent can take, such as interacting with a 3rd-party API. Tools all have a name, description, and a set of inputs (required and optional) for the function to run. Here is a simplified example of a tool that Paragon’s ActionKit API provides.

"name": "SALESFORCE_CREATE_RECORD_LEAD",
"description": "Use this tool when a user wants to Create a Lead in Salesforce",
     "parameters": {
           "properties": {
                "Id": {
                  "type": "string",
                  "description": "Lead ID"
                },
                "LastName": {
                  "type": "string",
                  "description": "Last Name"
                },
                "FirstName": {
                  "type": "string",
                  "description": "First Name"
                },
                "Title": {
                  "type": "string",
                  "description": "Title"
                },
                ...
                ..
              }
           "required": [
                "LastName",
                "Name",
                "Company"
              ],

Tool calling (sometimes called function calling or agentic actions) is the mechanism in which an agent uses a tool.

At its core, tool calling enables an AI agent to recognize when it needs to use an action to complete a task or retrieve data to answer a question, select the appropriate tool for the job, and execute actions using that tool.

This can happen both in a chat interface, or behind the scenes within the business logic of your AI product.

How does it actually work?

There are a few key components you need to understand about tool calling.

1. Tool availability: First, you need to give your AI agent the list of tools it can access. They ultimately need to work within the constraints of the tools they have access to.

2. Recognition & Selection: Your AI agent will reason about when a tool (or multiple tools) could be relevant in helping it answer a question or complete a task, based on the descriptions, names, and inputs provided alongside each tool.

3. Tool Execution: The underlying code for the tool is executed (such as an API request), and will return a response.

4. Response: Your AI agent will take the available information it gets from the prompt to use as inputs for calling the tool(s).

Let's break this down with a simple example from customer support software:

User: "What’s the latest status on our ticket with ACME?"

In this scenario, the AI agent would:

• Recognize it needs current ticket data

• Select a ticketing sysetem query tool

• Run the custom code that calls the Zendesk API to extract ticketing data

• Format and present the results to the user

Real-life examples

We used a support agent use case above to illustrate the tool calling flow, but let’s use a few real life examples of agent tool calling.

Intercom

Their AI support agent product, Fin, can query live external data such as Shopify for inventory + order status, to help answer questions from their users’ customers.

They also provide users the ability to define their own custom tools with other 3rd-party APIs, although this requires the user to configure the authentication, the API endpoints to hit, the inputs, as well as the tool descriptions.

OpenAI

While not publicly available at the time of writing this, OpenAI teased an AI sales agent with tool calling capabilities such as GET_CALENDAR_AVAILABILITY and SEND_GMAIL_EMAIL in a demo they held in Japan. This enables their AI sales agent to automatically email and schedule calls with prospects on behalf of sales reps.

tl;dv

tl;dv is an AI meeting assistant that can automatically turn meeting transcriptions into actionable next steps. With tool calling, not only can they generate a text summary of next steps, but they can also automatically:

• Update users’ CRMs with details from a sales call

• Create tasks in project management tools based on action items

• Send users a summary in Slack

Where should you start?

As with any other product feature, it’s all about understanding what problems your users are facing, and solving it in the most elegant way possible.

Tool calling opens up the possibility for two buckets of use cases:

• Querying external data

• Taking action in an external app

To discover where it can make sense for your product, look at your users’ workflow and the external actions they take while using your product. Maybe they need to look up some customer data in their CRM, send an email to a colleague, or save some data in their file storage system.

With tool calling, you can automate many of these actions and keep users locked into your application.

User experience considerations

The reason many AI products have struggled to gain adoption is because there is a lack of trust/confidence in their ability to deliver a satisfactory output. This is especially true when products try to ‘one shot’ the problem and miss the mark.

However, if you design your AI product/feature like an employee that reports to your users, you can mitigate a lot of these issues. Here are a few mechanisms of doing just that:

Human-in-the-loop

It’s always safer to have a user confirm a tool call, especially if the tool updates 3rd-party data. For example, if your AI agent can overwrite customers’ CRM data or send emails on their behalf, your customers will likely want oversight and an approval step before your AI agent performs these types of action.

This user confirmation flow is referred to as human-in-the-loop.

Agent reasoning/logs

It’s important that your users understand how your AI agent ended up with the output it provided, especially for more complex tasks.

This generally consists of two types of ‘logs’.

1. What steps/intermediary logic did your AI agent take to arrive at the end result?

2. What tools and corresponding inputs did it use?

To illustrate the first type of log, here’s an example with OpenAI’s Deep Research model. On the right, you can see the steps OpenAI’s Deep Research model took to arrive at the research report it generates, including the actions it took (such as search action and browse action).

For certain use cases or personas, providing visibility into the specific inputs/outputs of every tool call may be important.

Here’s an example of our playground for ActionKit, where our AI agent displays all the inputs and output payloads of every action it took as a result of a prompt.

Error Handling Since tool calling requires correct tool inputs from your AI agent as well as a functioning 3rd-party API, errors can and will happen. There are generally two types of errors:

1. Successful but inaccurate tool use

   • Tool selection error: Your AI agent uses the wrong tool

   • Tool inputs error: Your AI agent uses the right tool, but uses the wrong inputs (ie. sends an email, but to the wrong person)

2. Tool execution error: Your AI agent uses the right tool, and the right inputs, but the underlying function/request failed due to API errors.

From a product experience perspective, you need to design for these error cases.

When it comes to the first type of error, it may look like providing a way for recoverable errors to be retried with human intervention. For example, you could provide users a way to prompt the AI agent with additional guidance. This is also why showing the steps the agent took behind the scenes via logs is valuable.

However, when it comes to tool execution errors, it’s important to provide user-readable errors that help users understand what went wrong. This can involve passing through the underlying error from the 3rd-party side directly to the user, or having a separate AI agent turn the underlying error into something that your users can understand. Here’s an example from a sales & marketing automation product called Default.

Tool calling optimization

LLMs can behave in weird and unpredictable ways when it comes to tool calling, similar to how they may hallucinate in a chat experience. We mentioned ‘inaccurate tool use’ as a class of errors in the previous section - this is more common than you may think, even with optimized function tool descriptions.

As a result, there’s a lot of work that goes into tool calling optimization, which in very broad terms means ‘ensuring your AI agent does the thing it’s expected to do’.

We won’t get into the specifics here, but our developer advocate ran a deep evaluation on different ways to optimize tool calling - you can check it out here.

Conclusion

Tool calling is critical for the adoption of agentic AI products, and represents a fundamental shift in how users will interact with software in the future. Rather than navigating complex UIs and traversing across dozens of applications, we will arrive at the point where users will interact with agents and let the AI agents handle the execution details behind the scenes. This will enable your product to automate work for your users while building a center of gravity around your product, as they won’t need to go into other applications to complete their workflows.

If you’re looking to implement tool calling into your product with 3rd-party APIs, check out ActionKit. It’s a single API that will instantly give your AI agent access to query and write data across dozens of your users’ third-party applications like CRMs, ticketing, task management, and productivity (email, Slack, etc.).