In this experiment, a developer built the exact same Python API project 15 times using five different AI agent configurations (three runs each) to measure execution time and token usage. The goal was to isolate how different AI operational structures affect efficiency and performance using the MIMO 2.5 model. The benchmark project was deliberately kept simple: a three-endpoint Python API with API key authentication and open telemetry instrumentation.

The Five Experimental Approaches

• 1. Plan + Build (Baseline): Two sequential agents (Plan mode, then Build mode). This setup was highly consistent, averaging 4.5 minutes and 26,500 tokens.
• 2. YOLO Mode (Baseline): A single Build agent handling both planning and implementation. This was the fastest and cheapest run, averaging just over 4 minutes and 25,000 tokens.
• 3. Manual Specialist Agents: Five specialized agents (Architect, Security, Observability, Testing, and Documentation) toggled manually by the user. This approach performed poorly, taking 28 minutes and consuming 195,000 tokens. It also suffered from “identity drift,” where agents absorbed traits from the chat history and failed their specific tasks (e.g., the Build agent refusing to write code, claiming it was only a Documentation agent).
• 4. Orchestrated Primary Agents: The same specialists managed by an “Orchestrator” agent. The Orchestrator coordinated workflows and even dynamically generated a developer agent when it noticed a skills gap. This completed in 8 to 9 minutes, using 36,000 tokens.
• 5. Orchestrated Sub-Agents: Similar to the fourth approach, but utilizing native sub-agent modes. While it removed manual steps, it introduced minor token overhead (due to delegation and status-tracking protocols).

Key Findings and Takeaways

• Beware of Identity Drift: In long, single-context threads, specialized agents tend to lose their defined roles and match the conversational history. Developers should write explicit instructions detailing not only what an agent should do, but also what it should not do.
• The Cost of Orchestration: Every layer of coordination adds time and token overhead. For simple tasks, complex multi-agent frameworks are highly inefficient compared to a single, capable developer agent.
• Emergent Orchestrator Behaviors: While heavier on resources, the Orchestrator demonstrated valuable autonomous capabilities, such as identifying team skill gaps and looping in specialists (like the Architect) at critical project milestones.

Conclusion

For basic development tasks, a single-agent (“YOLO”) workflow remains the most economical and fastest choice. Multi-agent orchestration is powerful and shows promising emergent behaviors, but its high resource demands and latency mean it should be reserved for larger, highly complex systems where cross-functional review is genuinely required.

Mentoring question

How do you evaluate whether a development task is complex enough to justify the token cost and time overhead of a multi-agent orchestration framework, rather than relying on a simpler single-agent setup?

Source: https://youtube.com/watch?v=G-6rIOy3UBA&is=abDD25vkCtG2MmU_