AI Agents: Tools, Planning, and Autonomy
AI agents extend language models with tools, memory, and planning loops — moving from passive question-answering toward systems that pursue goals across long, multi-step tasks in real environments.
Key facts
- Agents combine perception, decision, action, memory, and planning.
- Function calling is now standard across all frontier LLMs.
- Reliability degrades multiplicatively with task length.
- Prompt injection is the dominant agent security failure mode.
- SWE-bench Verified scores by frontier agents exceeded 70% in 2025.
What Is an AI Agent?
An agent perceives, decides, and acts in an environment to pursue goals. Modern LLM-based agents combine a reasoning core (the LLM), tool use (function calling), memory (short-term context plus long-term retrieval), and an explicit or implicit planning loop.
The ReAct pattern (Yao et al., 2022) — interleaving Reasoning and Acting steps — became the canonical scaffolding. Modern frameworks include OpenAI's Agents SDK, Anthropic's Claude Agent SDK, LangGraph, and AutoGen.
Tools, Browsing, and Computer Use
Function calling lets the LLM emit structured calls to external APIs — search, databases, calculators, code execution. Web browsing extends knowledge beyond the training cutoff.
Computer-use agents (Anthropic Claude computer use, 2024; OpenAI Operator, 2025) control a virtual mouse, keyboard, and screen the way a human would, enabling automation of arbitrary GUI applications.
Why Agents Are Hard
Reliability degrades sharply with task length. Errors compound across steps, planning failures cascade, and recovery from off-trajectory states is unreliable.
Prompt injection — adversarial instructions embedded in retrieved data or web pages — is the dominant security failure mode for agents with tool access. Sandboxing, capability constraints, and human-in-the-loop confirmation are standard mitigations.
- Error accumulation: a 95% per-step success rate is only ~36% over 20 steps.
- Prompt injection via untrusted inputs (web, email, files).
- Cost: long agent runs can consume millions of tokens per task.
- Evaluation: benchmarks like SWE-bench, OSWorld, WebArena, GAIA measure real-world agentic capability.
State of the Art in 2026
SWE-bench Verified (real GitHub issues): frontier agents resolve >70% as of 2025. OSWorld and WebArena (general computer use): solid double-digit progress year-over-year. GAIA (general assistant): top systems at ~75% on level-1 questions.
Coding agents (Cursor Agents, Claude Code, Devin) are the most mature deployment category; general-purpose autonomous agents remain narrower than marketing implies.
Frequently asked
Are agents ready for production?
+
For narrow, bounded tasks with human review — coding, research, data extraction — increasingly yes. For open-ended high-stakes autonomy, no.
What is computer-use?
+
A capability allowing an AI agent to control a computer (mouse, keyboard, screen) the way a human would, enabling automation of arbitrary GUI applications without API access.
What is prompt injection?
+
An attack where adversarial instructions embedded in untrusted data (web pages, emails, documents) hijack an agent's behavior. It is the most widely-exploited LLM security issue.
Sources & further reading
Continue in this series
Foundations
Machine Learning: The Foundations
Neural Networks
Deep Learning: Hierarchical Representation from Raw Data
Architecture
The Transformer Architecture
LLMs
Large Language Models: How They Work and Where They Fail
Cross-Modal
Multimodal AI: Text, Vision, Audio, Video, and Action
Learning from Reward
Reinforcement Learning: From AlphaGo to RLHF