Auto Research is an open-source project created by legendary AI researcher Andrej Karpathy that enables artificial intelligence to improve itself autonomously. At its core, it is a recursive self-improvement loop where an AI agent continuously runs experiments, measures the outcomes against a specific metric, and keeps only the modifications that improve the result while discarding those that do not. This concept essentially automates the execution of iterative problem-solving and optimization.

The Three-File Architecture

The system relies on a strict, three-part structural design to ensure the AI optimizes effectively without “cheating” the test:

• program.md: The human-configured file that sets the overarching goals, constraints, and rules for the AI agent.
• The target file (e.g., train.py): The single file the AI agent is allowed to modify during its experiments (whether it contains code, configuration, a prompt, or a math equation).
• The evaluation script (e.g., prepare.py): The file that defines and measures the target metric. The AI is strictly prohibited from altering this file to prevent it from artificially manipulating its own score.

Key Conditions and Limitations

For an Auto Research loop to function properly, three criteria must be met: there must be a single, crystal-clear metric; the evaluation process must be fully automated (a human-in-the-loop makes it too slow); and the agent must be restricted to editing just one file. Additionally, experiments must operate within a fixed time budget to ensure the AI evaluates raw ideas rather than simply brute-forcing results by running longer. The system fails in domains where “better” is highly subjective—such as brand design, user experience, or subjective pricing—because the AI requires a hard numerical baseline to guide its decisions.

Applications Beyond Machine Learning

While originally designed to optimize machine learning models, Auto Research extends far beyond AI training. It can be applied to virtually any domain with a measurable outcome. Practical use cases include testing trading strategies (e.g., optimizing for the highest Sharpe ratio), automating marketing A/B tests (improving ad copy, YouTube titles, or email conversion rates), optimizing website load speeds, and refining prompt engineering for AI systems. As AI execution costs drop toward zero, the true human value will shift to defining the right metrics and system constraints.

Conclusions and Future Vision

Auto Research represents the tangible beginning of recursive self-improvement in AI systems. Karpathy’s long-term vision resembles projects like SETI@home, where a massive distributed network of AI agents operates across thousands of computers, autonomously running experiments to achieve breakthroughs across countless industries. By open-sourcing this foundational mechanism, the barrier to entry has been lowered, allowing developers to build autonomous, self-optimizing business and technical loops today.

Mentoring question

What is one measurable, repeatable process in your current business or daily workflow that could be transformed into an autonomous Auto Research loop?

Source: https://youtube.com/watch?v=uBWuKh1nZ2Y&is=jEwgnmzu_Ot6xNxq