Project Pilot: Assessing AI Model Capabilities in Autonomous Drone Flight

TL;DR

Anthropic and Andon Labs developed Project Pilot and the accompanying Drone-Bench benchmark to evaluate whether AI models can autonomously control drones for locate-and-follow surveillance tasks. The research reveals that while frontier models have nearly mastered target detection and following, they are still bottlenecked by their inability to accurately reconstruct 3D environments from video, though capabilities are progressing rapidly toward full autonomy.

Drone-Bench: Evaluating Autonomous Surveillance

Drone-Bench is a specialized benchmark created by Andon Labs to determine if AI agents can control a quad-rotor drone to locate and follow a specific person within an indoor office environment. The task is designed to reflect "dual-use" capabilities—technologies that have legitimate applications in search and rescue or disaster response but are also susceptible to abuse for unauthorized surveillance.

To evaluate end-to-end success, the benchmark decomposes the primary objective into five essential sub-tasks:

• Reconstruct: Converting office videos into a 3D model and generating a 2D obstacle map.
• Localize: Matching the drone's current view to the 2D obstacle map to determine its position.
• Navigate: Planning and flying a path between rooms while continuously correcting for noisy controls via localization.
• Localize: Matching the drone's current view to the 2D obstacle map to determine its position.
• Detect: Identifying a target person from a reference photo and providing a bounding box in the video feed.
• Follow: Using bounding boxes to maintain a stable distance and keep the target centered in view.

Andon Labs established a performance baseline using algorithms developed by human-AI teams. A model is considered to have completed a task if it meets or exceeds this baseline.

Model Performance and the "Reconstruct" Bottleneck

Andon Labs tested 15 models, including versions of GPT-4o, o1, o3, Claude Opus 4, Gemini 2.5 Pro, GPT-5, Gemini 3.1 Pro, Opus 4.5, GPT-5.2, Opus 4.7, GPT-5.5, Opus 4.8, Fable 5, and GPT-5.6 Sol. The findings indicate a steady increase in capability over time, with most models now reaching the baseline in four of the five sub-tasks.

Key Findings on Capabilities

• Top Performer: Claude Fable 5 was the best-performing model, surpassing the baseline in all tasks except reconstruction.
• Primary Bottleneck: The "Reconstruct" sub-task remains the most significant hurdle, with performance lagging at approximately 47% of the baseline by mid-2026.
• Failure Modes: In real-world demonstrations, Fable 5 excelled at detecting and following targets but failed at autonomous navigation between rooms, occasionally flying into walls because it could not accurately reconstruct the room layout.

Emergent Problem-Solving Behaviors

Researchers observed Fable 5 performing local analysis to improve its implementation. For example, the model analyzed grout lines on a simulated floor to estimate the drone's camera tilt (extrinsics) to within four degrees of the true value by extrapolating to a vanishing point. In other instances, it built its own internal 2D top-down reconstruction of the environment to test and iterate on its code before submission.

Reliability vs. Peak Capability

There is a significant gap between a model's best one-off performance and its consistent reliability. While frontier models may reach the human baseline in at least one out of ten simulations, their average performance often trails their peak capability by approximately six months.

Specifically, Fable 5's average performance today is roughly equivalent to the one-off best performance of previous models from the beginning of 2026. This suggests that while the "frontier" of capability is moving forward, the reliability required for autonomous operation lags behind.

Implications for AI Governance and Safety

Project Pilot highlights the risk associated with combining commercial-off-the-shelf (COTS) hardware with AI-tailored software. Anthropic notes a parallel between agentic coding—where humans initially approved every tool call but now trust models with long-horizon tasks—and the control of physical hardware.

As models pass reliability thresholds, the research warns that there will be increased pressure to remove human oversight to gain efficiency. Anthropic argues that because these capabilities implicate physical security and individual privacy, human oversight must be a deliberate safety requirement rather than a cost to be minimized.