Study measures failures in MCP agents
Researchers introduced MCPHallu, a benchmark for diagnosing failures in agents that use tools through the Model Context Protocol (MCP).
- MCPHallu contains 358 tasks across 36 isolated MCP servers.
- The study evaluated 14 agents from eight providers over 5,012 trajectories.
- On tasks with unreachable goals, 23.2% of runs passed.
Context and analysis by Noosigma AI
Editorial summary and context based on the cited source. Claims and results attributed to the organization are reports from that source; they do not constitute independent validation by Noosigma AI.
The suite contains 358 tasks across five capability areas and runs on 36 production MCP servers in isolated containers. Each task is designed to highlight risks such as unreachable goals, tool misuse, context forgetting or reasoning failures.
Why it matters
The authors evaluated 14 agents from eight providers over 5,012 trajectories. The average score was 0.642 on a scale from zero to one; on unreachable-goal tasks it fell to 0.458, and only 23.2% of runs passed. The study reports that many agents kept calling tools or claimed completion instead of recognizing that a goal was infeasible.
The paper is original research posted on arXiv, and its figures apply to the benchmark and configurations tested, not to every agent in use. Source: arXiv, November 5, 2026.