The Mental Model: Tools Are Structured Output with Side Effects
At its core, function calling is just the model outputting JSON instead of text. Instead of saying "I looked up the weather and it's 72 degrees," the model outputs:
{
"tool": "get_weather",
"arguments": {
"location": "San Francisco",
"units": "fahrenheit"
}
}
Your code executes the function, gets the result, and feeds it back to the model. The model then generates a natural language response incorporating the tool result.
A critical distinction: the model does not "call" anything. It generates text that looks like a function call. Your orchestration code parses that text, executes the real function, and manages the conversation loop. The model is a JSON generation engine with really good intent detection. Every design decision should flow from this understanding.
Tool Schema Design: Where 80% of Reliability Comes From
The single biggest lever for tool-call reliability is the schema. A well-designed schema constrains the model's output space, making correct calls more likely and hallucinated calls harder.
// Bad: too much freedom
const badTool = {
name: "manage_subscription",
description: "Manage a user's subscription",
parameters: {
type: "object",
properties: {
action: { type: "string", description: "What to do" },
details: { type: "string", description: "Additional details" },
},
},
};
// Good: constrained, explicit, with reasoning
const goodTool = {
name: "update_subscription",
description: "Change a user's subscription plan. Only call this when the user has explicitly requested a plan change and confirmed the new plan.",
parameters: {
type: "object",
properties: {
reason: {
type: "string",
description: "Why you are changing this subscription (quote the user's request)",
},
user_id: {
type: "string",
description: "The user's ID from the conversation context",
pattern: "^usr_[a-zA-Z0-9]{20}$",
},
new_plan: {
type: "string",
enum: ["starter", "professional", "enterprise"],
description: "The target plan",
},
billing_cycle: {
type: "string",
enum: ["monthly", "annual"],
description: "Billing frequency for the new plan",
},
effective_date: {
type: "string",
enum: ["immediate", "next_billing_cycle"],
description: "When the change takes effect",
},
},
required: ["reason", "user_id", "new_plan", "billing_cycle", "effective_date"],
},
};
Key patterns:
- Enums over free text: Every field that has a known set of valid values should be an enum. The model picks from a list instead of generating arbitrary strings.
- Reason field: Forces the model to articulate its reasoning before the action parameters. This acts like chain-of-thought prompting but within the structured output.
- Regex patterns: For IDs, emails, dates, constrain the format so obviously wrong values are rejected at the schema level.
- Descriptive constraints: "Only call this when the user has explicitly requested..." in the tool description gives the model clear criteria for when to use the tool.
The Orchestration Loop: Beyond Single-Shot
Real agents don't make one tool call. They make chains of calls, sometimes in parallel, with branching logic based on results. Here is a proven pattern:
async function runAgent(messages, tools, maxIterations = 10) {
let iterations = 0;
while (iterations < maxIterations) {
iterations++;
const response = await callModel({
messages,
tools,
tool_choice: iterations === 1 ? 'auto' : 'auto',
});
// If the model wants to respond with text, we're done
if (response.stop_reason === 'end_turn') {
return {
response: response.content,
toolCalls: messages.filter(m => m.role === 'tool_call'),
};
}
// Process tool calls
if (response.stop_reason === 'tool_use') {
const toolCalls = response.tool_calls;
// Add the assistant's message (with tool calls) to history
messages.push({
role: 'assistant',
content: response.content,
tool_calls: toolCalls,
});
// Execute tools, parallel when safe, sequential when dependent
const toolResults = await executeToolCalls(toolCalls, {
parallel: canRunInParallel(toolCalls),
});
// Add each tool result to the conversation
for (const result of toolResults) {
messages.push({
role: 'tool',
tool_call_id: result.id,
content: JSON.stringify(result.output),
});
}
// Continue the loop, the model will see the results and decide what's next
continue;
}
// Unexpected stop reason
throw new AgentError(`Unexpected stop reason: ${response.stop_reason}`);
}
throw new AgentError(`Agent exceeded maximum iterations (${maxIterations})`);
}
The maxIterations guard is critical. Without it, a confused model can loop forever, calling a tool, getting an error, trying a slightly different variation, getting the same error, ad infinitum. Agents have been observed burning through thousands of API calls in minutes without this guard.
Parallel Tool Calls: When the Model Gets Ambitious
Modern models can request multiple tool calls in a single response. This is powerful for things like "look up the user's profile AND their recent orders", two independent lookups that can happen simultaneously.
async function executeToolCalls(toolCalls, options = {}) {
// Classify calls as parallelizable or sequential
const { parallel, sequential } = classifyToolCalls(toolCalls);
const results = [];
// Run parallel calls concurrently
if (parallel.length > 0) {
const parallelResults = await Promise.allSettled(
parallel.map(call => executeSingleTool(call))
);
for (let i = 0; i < parallelResults.length; i++) {
const result = parallelResults[i];
if (result.status === 'fulfilled') {
results.push({
id: parallel[i].id,
output: result.value,
});
} else {
results.push({
id: parallel[i].id,
output: {
error: true,
message: result.reason.message,
retryable: isRetryable(result.reason),
},
});
}
}
}
// Run sequential calls in order
for (const call of sequential) {
try {
const output = await executeSingleTool(call);
results.push({ id: call.id, output });
} catch (err) {
results.push({
id: call.id,
output: {
error: true,
message: err.message,
retryable: isRetryable(err),
},
});
}
}
return results;
}
function classifyToolCalls(toolCalls) {
const writeTools = new Set(['update_subscription', 'create_order', 'send_email']);
const parallel = [];
const sequential = [];
for (const call of toolCalls) {
if (writeTools.has(call.function.name)) {
sequential.push(call); // Mutations go sequential
} else {
parallel.push(call); // Reads can parallelize
}
}
return { parallel, sequential };
}
The read/write classification is important. Two simultaneous reads are fine. Two simultaneous writes to the same resource can cause race conditions. And a read followed by a conditional write must be sequential.
Error Handling: Teaching the Model to Recover
When a tool call fails, you need to tell the model what happened in a way that helps it recover:
async function executeSingleTool(call) {
const tool = toolRegistry.get(call.function.name);
if (!tool) {
// Don't throw, return a helpful error the model can understand
return {
error: true,
message: `Unknown tool "${call.function.name}". Available tools: ${[...toolRegistry.keys()].join(', ')}`,
};
}
let args;
try {
args = JSON.parse(call.function.arguments);
} catch (e) {
return {
error: true,
message: `Invalid JSON in arguments: ${e.message}. Please format arguments as valid JSON.`,
};
}
// Validate arguments against schema
const validation = validateArgs(tool.schema, args);
if (!validation.valid) {
return {
error: true,
message: `Invalid arguments: ${validation.errors.join('; ')}`,
hint: `Expected schema: ${JSON.stringify(tool.schema.properties)}`,
};
}
try {
const result = await tool.execute(args);
return { success: true, data: result };
} catch (err) {
if (err instanceof NotFoundError) {
return {
error: true,
message: `${args.id} was not found. It may have been deleted or the ID may be incorrect.`,
suggestion: 'Try searching for the resource by name instead of ID.',
};
}
if (err instanceof PermissionError) {
return {
error: true,
message: `Access denied: ${err.message}. The current user does not have permission for this operation.`,
suggestion: 'Inform the user that this action requires elevated permissions.',
};
}
if (err instanceof RateLimitError) {
return {
error: true,
message: 'Rate limit exceeded. Please wait before retrying.',
retryable: true,
retryAfter: err.retryAfter,
};
}
// Unknown errors, don't expose internals
logger.error('Tool execution failed', { tool: call.function.name, args, error: err });
return {
error: true,
message: 'An internal error occurred while executing this tool. Please try a different approach or ask the user for more information.',
};
}
}
The essential principle: error messages are prompts. The model reads them and decides what to do next. "An error occurred" gives the model nothing to work with. "User not found, try searching by name instead of ID" gives it a recovery path.
Structured Output: Guaranteed Schema Compliance
For critical operations, you want the model's output to be guaranteed valid, not just probably valid:
// OpenAI strict mode
const response = await openai.chat.completions.create({
model: "gpt-4o",
messages: [...],
tools: [{
type: "function",
function: {
name: "create_invoice",
description: "Create an invoice for a customer",
strict: true, // Guarantees schema compliance
parameters: {
type: "object",
properties: {
customer_id: { type: "string" },
line_items: {
type: "array",
items: {
type: "object",
properties: {
description: { type: "string" },
quantity: { type: "integer" },
unit_price_cents: { type: "integer" },
},
required: ["description", "quantity", "unit_price_cents"],
additionalProperties: false,
},
},
due_date: { type: "string" },
currency: { type: "string", enum: ["USD", "EUR", "GBP"] },
},
required: ["customer_id", "line_items", "due_date", "currency"],
additionalProperties: false,
},
},
}],
});
With strict mode, the JSON structure is guaranteed to match. No missing fields, no extra fields, no wrong types. The model's constrained decoding ensures it. However, the values can still be wrong. The model might output "customer_id": "cust_abc123" when the actual customer ID is "cust_xyz789". Schema compliance is not semantic correctness.
The Confirmation Pattern: Humans in the Loop
For high-stakes actions (money, data deletion, external communications), never let the agent execute directly. Use a confirmation step:
const DANGEROUS_TOOLS = new Set([
'send_email', 'process_refund', 'delete_account',
'update_billing', 'create_order',
]);
async function executeWithConfirmation(call, context) {
if (!DANGEROUS_TOOLS.has(call.function.name)) {
return executeSingleTool(call);
}
// Instead of executing, return a confirmation request
return {
requires_confirmation: true,
tool: call.function.name,
arguments: JSON.parse(call.function.arguments),
summary: await generateHumanReadableSummary(call),
confirmation_token: generateToken(call), // signed, time-limited
};
}
// The model sees this and tells the user:
// "I'd like to process a $49.99 refund to order #12345. Should I go ahead?"
// User confirms → your code validates the token and executes
This pattern is non-negotiable for production agents. No matter how good the model gets, there will always be ambiguous situations where a human should decide. The agent's job is to prepare the action, not to authorize it.
Retry Strategies That Actually Work
class ToolCallRetryPolicy {
constructor(options = {}) {
this.maxRetries = options.maxRetries || 2;
this.retryableErrors = options.retryableErrors || [
'rate_limit', 'timeout', 'temporary_failure',
];
}
async executeWithRetry(call, executeFunction) {
let lastError;
for (let attempt = 0; attempt <= this.maxRetries; attempt++) {
try {
const result = await executeFunction(call);
if (result.error && result.retryable && attempt < this.maxRetries) {
lastError = result;
const backoff = Math.min(1000 * Math.pow(2, attempt), 10000);
await sleep(backoff);
continue;
}
return result;
} catch (err) {
lastError = err;
if (!this.isRetryable(err) || attempt >= this.maxRetries) {
throw err;
}
const backoff = Math.min(1000 * Math.pow(2, attempt), 10000);
await sleep(backoff);
}
}
return {
error: true,
message: `Tool call failed after ${this.maxRetries + 1} attempts: ${lastError.message}`,
};
}
}
Important: don't retry model calls when a tool fails. Feed the error back to the model and let it decide what to do. The model might choose to try a different tool, ask the user for clarification, or give up gracefully. Blindly retrying the same tool call with the same parameters rarely helps.
Observability: What to Log
Every production agent needs comprehensive logging. Here's the minimum:
function logToolCall(call, result, metadata) {
const logEntry = {
timestamp: new Date().toISOString(),
conversation_id: metadata.conversationId,
turn_number: metadata.turnNumber,
model: metadata.model,
// The full model response that triggered this call
model_response_id: metadata.responseId,
// Tool call details
tool_name: call.function.name,
tool_arguments: JSON.parse(call.function.arguments),
tool_call_id: call.id,
// Result
success: !result.error,
result_summary: summarizeResult(result),
execution_time_ms: metadata.executionTimeMs,
// Context
user_id: metadata.userId,
session_id: metadata.sessionId,
};
// Structured log, ship to your observability stack
logger.info('tool_call', logEntry);
// Metrics
metrics.increment('tool_calls_total', {
tool: call.function.name,
success: String(!result.error),
});
metrics.histogram('tool_call_duration_ms', metadata.executionTimeMs, {
tool: call.function.name,
});
}
When something goes wrong at 2 AM, and it will, the ability to reconstruct the entire conversation: what the user said, what the model decided, what tools it called, what results it got, and what it said next. Without this, debugging agent failures is guesswork.
The Routing Pattern: Scaling Beyond 20 Tools
When you have more tools than a model can reliably choose from, use a two-stage approach:
const TOOL_CATEGORIES = {
billing: ['get_invoices', 'create_refund', 'update_payment_method', 'get_subscription'],
support: ['create_ticket', 'get_ticket_status', 'escalate_ticket', 'close_ticket'],
account: ['get_profile', 'update_profile', 'reset_password', 'get_login_history'],
product: ['search_products', 'get_product_details', 'check_inventory', 'get_pricing'],
};
// Stage 1: Route to category
const routerTools = [{
name: "select_tool_category",
description: "Select the most relevant tool category for the user's request",
parameters: {
type: "object",
properties: {
category: {
type: "string",
enum: Object.keys(TOOL_CATEGORIES),
},
reason: { type: "string" },
},
required: ["category", "reason"],
},
}];
// Stage 2: Use category-specific tools
async function routedAgent(messages) {
// First, determine the category
const routingResponse = await callModel({
messages,
tools: routerTools,
tool_choice: { type: "function", function: { name: "select_tool_category" } },
});
const category = routingResponse.tool_calls[0].function.arguments.category;
const relevantTools = TOOL_CATEGORIES[category].map(name => toolRegistry.get(name));
// Then, run the agent with only the relevant tools
return runAgent(messages, relevantTools);
}
This two-stage pattern keeps each model call focused on a small set of tools, dramatically improving selection accuracy. The overhead of the extra API call is negligible compared to the cost of a wrong tool selection.
Key Lessons from Production Agent Development
-
Schema design is your primary lever. Spend more time designing tool schemas than writing orchestration code. Constrained enums, required reason fields, and clear descriptions do more for reliability than any retry logic.
-
The model is not your executor, it's your planner. Don't give the model tools that do atomic, irreversible actions. Give it tools that prepare actions, and let your code handle execution with proper safeguards.
-
Error messages are prompts. Every string your tool returns is a prompt to the model. Write error messages as instructions for recovery, not as developer debug output.
-
Parallel tool calls are a feature, not a bug. Design your tools to be parallelizable by default. Read operations should always be safe to parallelize. Mutations should be explicit about their dependencies.
-
Log everything. The first time you debug a production agent issue without full conversation logs, the value of logging becomes unmistakable.
-
The max iteration guard is not optional. Models in loops can burn through your API budget in minutes. Always cap iterations and always have a cost circuit breaker.
The tooling is improving fast. Structured output modes make schema compliance guaranteed. Multi-turn conversation handling is getting more reliable. But the fundamental challenge remains: you're building a system where a probabilistic text generator makes decisions about deterministic operations. Respect that gap, design for it, and your agents will be the ones that actually work in production.
Reproducible evaluation: hostile instructions and permissions
Agent Injection Evals crosses plain or instruction-isolated prompts with permissive or task-scoped tool policies. It uses the same pinned local model and preserves all 120 evaluation generations, including malformed outputs and failed legitimate tasks.
Isolation allowed five unauthorized writes; combined protection attempted the same five and policy blocked them. No strategy completed any of the six clean tasks. Calling that zero attack rate a winning defense would hide the problem: the agent was not doing its job either. The project page separates capability, attempts, effects, blocked calls and resource use. These are simulated tools and a small synthetic set, not a security validation for arbitrary agents.