Lesson 16: Tool-Using Agents | AI Engineering
Module 4: AI Agents & Tools

Tool-Using Agents

Design tool registries, implement robust execution loops with error recovery, enforce sandboxing, and build agents that chain multiple tool calls to complete complex research tasks.

Lesson 16 ~25 min Advanced

Tool Definition Patterns

What Makes a Good Tool?

A tool is any function the agent can call. Good tools have: (1) a clear, unambiguous name, (2) a precise description the LLM can reason about, (3) a well-defined schema for inputs/outputs, and (4) deterministic behavior with clear error messages. The LLM reads descriptions to decide when and how to use each tool.

"""Tool Definition — The foundation of tool-using agents."""
from dataclasses import dataclass, field
from typing import Callable, Any
import json

@dataclass
class ToolParameter:
    name: str
    type: str  # "string", "integer", "boolean", "array"
    description: str
    required: bool = True
    enum: list = field(default_factory=list)

@dataclass
class Tool:
    name: str
    description: str
    parameters: list[ToolParameter]
    function: Callable[..., str]
    requires_confirmation: bool = False  # Human-in-the-loop
    timeout_seconds: int = 30
    
    def to_openai_schema(self) -> dict:
        """Convert to OpenAI function calling format."""
        properties = {}
        required = []
        for param in self.parameters:
            properties[param.name] = {
                "type": param.type,
                "description": param.description,
            }
            if param.enum:
                properties[param.name]["enum"] = param.enum
            if param.required:
                required.append(param.name)
        
        return {
            "type": "function",
            "function": {
                "name": self.name,
                "description": self.description,
                "parameters": {
                    "type": "object",
                    "properties": properties,
                    "required": required,
                },
            },
        }

# Example: Define a web search tool
search_tool = Tool(
    name="web_search",
    description="Search the web for current information. Use for facts, news, or data you don't know.",
    parameters=[
        ToolParameter("query", "string", "The search query"),
        ToolParameter("num_results", "integer", "Number of results (1-10)", required=False),
    ],
    function=lambda query, num_results=5: tavily_search(query, num_results),
    timeout_seconds=15,
)

Tool Selection Lifecycle

The lifecycle has distinct phases where different things can go wrong:

  • Selection: LLM picks the wrong tool → improve descriptions, add examples
  • Argument Generation: LLM passes wrong args → tighten schemas, validate
  • Execution: Tool fails (network, timeout, bad input) → retry with backoff
  • Result Interpretation: LLM misreads output → structure results clearly

Building a Tool Registry

"""Complete Tool Registry with validation, execution, and error handling."""
import time
import traceback
from concurrent.futures import ThreadPoolExecutor, TimeoutError

class ToolRegistry:
    """Central registry for all available tools."""
    
    def __init__(self):
        self.tools: dict[str, Tool] = {}
        self.execution_log: list[dict] = []
    
    def register(self, tool: Tool):
        """Register a tool, checking for name conflicts."""
        if tool.name in self.tools:
            raise ValueError(f"Tool '{tool.name}' already registered")
        self.tools[tool.name] = tool
    
    def get_schemas(self) -> list[dict]:
        """Get all tool schemas for the LLM."""
        return [tool.to_openai_schema() for tool in self.tools.values()]
    
    def get_tool_descriptions(self) -> str:
        """Human-readable tool list for system prompts."""
        lines = []
        for tool in self.tools.values():
            params = ", ".join(f"{p.name}: {p.type}" for p in tool.parameters)
            lines.append(f"- {tool.name}({params}): {tool.description}")
        return "\n".join(lines)
    
    def validate_args(self, tool_name: str, args: dict) -> tuple[bool, str]:
        """Validate arguments against tool schema."""
        tool = self.tools.get(tool_name)
        if not tool:
            return False, f"Unknown tool: {tool_name}"
        
        for param in tool.parameters:
            if param.required and param.name not in args:
                return False, f"Missing required parameter: {param.name}"
            if param.name in args and param.enum:
                if args[param.name] not in param.enum:
                    return False, f"Invalid value for {param.name}. Must be one of: {param.enum}"
        
        return True, "Valid"
    
    def execute(self, tool_name: str, args: dict, 
                confirm_fn: Callable = None) -> dict:
        """Execute a tool with timeout, validation, and logging."""
        start_time = time.time()
        
        # Validate
        valid, error = self.validate_args(tool_name, args)
        if not valid:
            return {"success": False, "error": error, "duration": 0}
        
        tool = self.tools[tool_name]
        
        # Human-in-the-loop confirmation
        if tool.requires_confirmation and confirm_fn:
            approved = confirm_fn(tool_name, args)
            if not approved:
                return {"success": False, "error": "User rejected execution", "duration": 0}
        
        # Execute with timeout
        try:
            with ThreadPoolExecutor(max_workers=1) as executor:
                future = executor.submit(tool.function, **args)
                result = future.result(timeout=tool.timeout_seconds)
            
            duration = time.time() - start_time
            log_entry = {
                "tool": tool_name, "args": args, "result": result,
                "success": True, "duration": duration,
            }
            self.execution_log.append(log_entry)
            return {"success": True, "result": result, "duration": duration}
        
        except TimeoutError:
            return {"success": False, "error": f"Tool timed out after {tool.timeout_seconds}s", 
                    "duration": tool.timeout_seconds}
        except Exception as e:
            return {"success": False, "error": f"{type(e).__name__}: {str(e)}", 
                    "duration": time.time() - start_time}

Agent Execution Loop with Error Recovery

"""Full agent loop using OpenAI function calling + tool registry."""
import openai

client = openai.OpenAI()

def run_tool_agent(query: str, registry: ToolRegistry, 
                   max_iterations: int = 10, max_retries: int = 2) -> str:
    """Agent loop with automatic error recovery."""
    
    messages = [
        {"role": "system", "content": f"""You are a helpful research assistant with access to tools.
Use tools to find information, then synthesize a comprehensive answer.
If a tool fails, try a different approach or rephrase your query.

Available tools:
{registry.get_tool_descriptions()}"""},
        {"role": "user", "content": query},
    ]
    
    tool_schemas = registry.get_schemas()
    
    for iteration in range(max_iterations):
        response = client.chat.completions.create(
            model="gpt-4o",
            messages=messages,
            tools=tool_schemas,
            tool_choice="auto",  # Let model decide
        )
        
        msg = response.choices[0].message
        messages.append(msg)
        
        # If no tool calls, we have our final answer
        if not msg.tool_calls:
            return msg.content
        
        # Process each tool call
        for tool_call in msg.tool_calls:
            fn_name = tool_call.function.name
            fn_args = json.loads(tool_call.function.arguments)
            
            # Execute with retries
            result = None
            for attempt in range(max_retries + 1):
                result = registry.execute(fn_name, fn_args)
                if result["success"]:
                    break
                # On failure, wait briefly before retry
                time.sleep(1)
            
            # Format result for the LLM
            if result["success"]:
                tool_response = result["result"]
            else:
                tool_response = f"ERROR: {result['error']}. Please try a different approach."
            
            messages.append({
                "role": "tool",
                "tool_call_id": tool_call.id,
                "content": str(tool_response),
            })
    
    return "Agent reached maximum iterations. Partial results may be in the conversation."

Chaining Tool Calls

Multi-Step Tool Chains

Real tasks require multiple tool calls where each depends on the previous result. The agent searches for something, reads the top result, extracts data, then calculates with it. The key challenge: the context window fills up fast. Summarize intermediate results to stay within budget.

"""Tools that chain: search → read → extract → synthesize."""

# Tool: Read a webpage and extract content
read_page_tool = Tool(
    name="read_webpage",
    description="Read the full text content of a webpage URL. Use after searching to get details.",
    parameters=[
        ToolParameter("url", "string", "The full URL to read"),
    ],
    function=lambda url: fetch_and_extract(url),
    timeout_seconds=20,
)

# Tool: Summarize long text to save context space
summarize_tool = Tool(
    name="summarize_text",
    description="Summarize a long text into key points. Use when text is too long to include directly.",
    parameters=[
        ToolParameter("text", "string", "The text to summarize"),
        ToolParameter("focus", "string", "What aspect to focus the summary on", required=False),
    ],
    function=lambda text, focus="": llm_summarize(text, focus),
    timeout_seconds=30,
)

# Tool: Write structured report
write_report_tool = Tool(
    name="write_report",
    description="Write a structured report from gathered research. Use as your final step.",
    parameters=[
        ToolParameter("topic", "string", "The report topic"),
        ToolParameter("findings", "string", "Key findings to include (as bullet points)"),
        ToolParameter("format", "string", "Report format", enum=["brief", "detailed", "executive"]),
    ],
    function=lambda topic, findings, format="detailed": generate_report(topic, findings, format),
    requires_confirmation=True,  # Confirm before generating long content
)

Security & Sandboxing

⚠️ Critical Security Warning: Tool-using agents execute code based on LLM outputs. LLMs can be manipulated via prompt injection — a malicious webpage could instruct the agent to call dangerous tools. Never give agents unsandboxed access to file systems, databases, or network calls without strict allowlists.
"""Sandboxing patterns for tool execution."""

class SandboxedToolRegistry(ToolRegistry):
    """Registry with security constraints."""
    
    def __init__(self, allowed_domains: list[str] = None,
                 blocked_tools: list[str] = None,
                 rate_limits: dict[str, int] = None):
        super().__init__()
        self.allowed_domains = allowed_domains or []
        self.blocked_tools = blocked_tools or []
        self.rate_limits = rate_limits or {}  # tool_name -> max calls per minute
        self._call_counts: dict[str, list[float]] = {}
    
    def execute(self, tool_name: str, args: dict, **kwargs) -> dict:
        # Check if tool is blocked
        if tool_name in self.blocked_tools:
            return {"success": False, "error": "Tool is disabled by security policy"}
        
        # Check rate limits
        if tool_name in self.rate_limits:
            now = time.time()
            calls = self._call_counts.get(tool_name, [])
            calls = [t for t in calls if now - t < 60]  # Last minute
            if len(calls) >= self.rate_limits[tool_name]:
                return {"success": False, "error": "Rate limit exceeded"}
            calls.append(now)
            self._call_counts[tool_name] = calls
        
        # URL allowlist check
        if "url" in args and self.allowed_domains:
            from urllib.parse import urlparse
            domain = urlparse(args["url"]).netloc
            if not any(domain.endswith(d) for d in self.allowed_domains):
                return {"success": False, "error": f"Domain '{domain}' not in allowlist"}
        
        return super().execute(tool_name, args, **kwargs)

# Usage
registry = SandboxedToolRegistry(
    allowed_domains=["wikipedia.org", "arxiv.org", "github.com"],
    rate_limits={"web_search": 10, "read_webpage": 20},
)
Security Layer What It Prevents Implementation
Input Validation Malformed arguments, injection Schema validation + sanitization
Domain Allowlist Accessing unauthorized sites URL parsing + allowlist check
Rate Limiting Resource exhaustion, cost runaway Sliding window counter
Timeout Hanging connections, infinite loops ThreadPoolExecutor + timeout
Human Confirmation Destructive or expensive actions Callback before execution
Output Sanitization Injection via tool results Strip control chars, limit length

Mini-Project: Research Agent

🛠️ Build a Research Agent with Tavily + Page Reading + Reports

Create a tool-using agent that can research any topic by searching the web, reading relevant pages, and generating a structured report.

  1. Set up the tool registry with three tools:
    • web_search(query) — using Tavily API for high-quality search
    • read_page(url) — fetch and extract text from URLs
    • write_report(topic, findings, format) — generate final output
  2. Implement sandboxing: domain allowlist, rate limits (10 searches/minute), 20s timeouts
  3. Build the agent loop using OpenAI function calling (not text parsing)
  4. Add error recovery: if search fails, try alternative query; if page fails, skip and try next
  5. Implement context management: summarize long page contents before adding to messages
  6. Test with research queries:
    • "Research the latest developments in quantum computing in 2024"
    • "Compare the top 3 Python web frameworks by performance and ease of use"
    • "What are the environmental impacts of large language model training?"
  7. Add a requires_confirmation=True step before the final report generation

Stretch Goals

  • Add a save_to_file tool that writes reports to disk (with confirmation)
  • Implement a "research plan" step where the agent outlines its search strategy first
  • Add citation tracking: each fact in the report links back to its source URL

Key Takeaways

  • Good tools have clear names, precise descriptions, and well-defined schemas — the LLM reads these to decide usage
  • A Tool Registry centralizes registration, validation, execution, timeout, and logging
  • Error recovery is essential: retry on transient failures, give the LLM error context to adapt
  • Security layers (allowlists, rate limits, confirmation, timeouts) are mandatory for production agents
  • Chain tool calls by letting the agent observe each result and decide the next step
  • Summarize long intermediate results to manage context window budget