Recent findings by Stanford researchers reveal that the orchestration code wrapping a language model—known as the “harness”—drives up to a six-fold difference in performance, proving more critical than the model itself. In modern AI, an agent is defined as the model plus its harness. If you aren’t building the underlying model, your primary job is engineering the harness.
The Operating System of Agents
The relationship between a Large Language Model (LLM) and its harness is akin to a CPU and an operating system. The raw model acts as a powerful but inert CPU, while the harness serves as the OS coordinating context windows (RAM), external databases (disk), and tool integrations (device drivers). This orchestration controls everything outside the model weights, including prompt chaining, memory management, and evaluator loops. Architecting these workflows correctly avoids common naive failure modes like “oneshotting” (exhausting context limits immediately) or premature completion.
The Shift to Natural Language Harnesses (NLH)
Researchers have discovered that representing harness logic in structured natural language, rather than traditional code like Python or YAML, can drastically improve outcomes. Frameworks separating runtime infrastructure from task-specific control logic enable clean, controlled experiments. Surprisingly, modular testing reveals that more structure is not always better. While self-evolution attempt loops consistently improve performance, forced verifiers or multi-candidate searches can actually degrade it. Simply migrating a desktop automation harness from native code to natural language improved benchmark performance from 30% to 47% while radically reducing required LLM calls and runtime.
Automating and Optimizing the Harness
If the design of the harness matters this much, can it be optimized automatically? Stanford’s Meta harness proves it can by treating the orchestration pipeline itself as an optimization target. By using an agentic proposer to diagnose raw execution traces of failures, the system automatically writes, tests, and refines new harnesses. This automated optimization allowed a smaller model to outrank larger models. Crucially, a harness optimized on one model successfully transferred to five others, proving that the reusable asset is the optimized harness, not the model.
Conclusions and The Craft of Subtraction
AI development has rapidly evolved from prompt engineering to context engineering, and has now entered the era of harness engineering. Interestingly, as underlying models improve, they outgrow their old orchestration crutches. Consequently, mature harness engineering often becomes a “craft of subtraction”—pruning away unnecessary structures rather than adding new ones. The ultimate takeaway is clear: investing in your orchestration harness yields larger, faster, and more reliable gains than waiting for the next model upgrade.
Mentoring question
As underlying AI models continue to improve, how can we apply the ‘craft of subtraction’ to our current AI workflows to simplify and optimize our orchestration harnesses?
Source: https://youtube.com/watch?v=Xxuxg8PcBvc&is=A4ZwMqoxCNZ67X8Z