Last week, while scrolling through Twitter—I know we are supposed to call it X now, but I am not used to it—I saw Andrej Karpathy share a new repository called autoresearch. I almost dismissed it as another framework. Then I read the description: start it before bed, wake up to hundreds of completed experiments and an improved model.

Naturally, I cloned it right away.

What Is It? #

The idea is simple, but the implementation is clever. Suppose you have training code for a GPT model. Normally, you tweak some hyperparameters, run an experiment, wait 20 minutes, inspect the result, decide to lower the learning rate, and start again. This loop can continue for hours.

autoresearch hands that loop to an AI agent. Whether you use Claude, Codex, or another coding agent, it reads the code, changes something, runs it, checks the result, and tries again. Continuously.

Each experiment takes exactly five minutes: twelve per hour, or more than a hundred overnight. In the time it takes me to write this article, you might run three or four experiments manually. The agent runs a hundred.

The Repository Surprised Me #

There are only 630 lines of code. Seriously. I expected a massive framework, but it is minimal: three files.

First, prepare.py downloads data, trains the tokenizer, and contains evaluation functions. Crucially, the agent cannot modify it. I will explain why shortly.

Second, train.py is the agent’s playground. It defines the GPT model, optimizer, and training loop. The agent can change the number of layers, experiment with attention, adjust the batch size, or modify anything else in that file.

Third, program.md is my favorite part: a human-written strategy document in Markdown. You tell the agent what to try, where to focus, and what to avoid. There is no code in it, just natural-language instructions. The original is in English, but you could write them in Turkish too.

Does It Actually Work? #

Karpathy shared results from roughly 700 experiments over two days. Most did not help, which is normal: research involves many failed experiments. About twenty produced meaningful improvements. GPT-2 training time fell from 2.02 hours to 1.80 hours, roughly eleven percent.

Eleven percent might not sound dramatic, but improvements like that can take researchers weeks to find. Getting them while sleeping changes the perspective.

Shopify CEO Tobi Lütke’s experience was even more striking. He ran autoresearch overnight for 37 experiments. The result was a 0.8-billion-parameter model outperforming a 1.6-billion-parameter model by nineteen percent. A model half the size beat the larger one. Perhaps optimizing existing models deserves as much attention as making them bigger.

This Is Not Limited to Machine Learning #

While exploring it, I realized the idea applies to almost any try–measure–repeat loop. Model training is the obvious example, but Eric Siu, the founder of Single Grain, discussed applying it to marketing experiments. He suggested that teams running thirty experiments a year today could run more than 36,500 in the next generation. I do not know whether that number is realistic, but the direction makes sense.

Consider website A/B testing: colors, button placement, or wording. A system could try variations automatically and measure which converts better. The principle is the same.

A Few Things I Keep Thinking About #

It is not all straightforward. Some questions deserve attention.

Is Five Minutes Enough?

Each experiment has a five-minute budget. The appeal is a consistent time limit: whether you use an H100 or an RTX 4090, the question is how much training you can accomplish within it. But some effects will not show up in five minutes. Overfitting might emerge after two hours, and some architectural changes matter only in longer training runs.

Karpathy is probably aware of this. It is a deliberate trade-off: faster experiments, but a shorter horizon.

Who Writes program.md?

This is the question I have thought about most. Why can Karpathy write an effective program.md? Because he has spent years training models, debugging exploding gradients at three in the morning, and learning what different hyperparameters do.

There is a paradox here. If repetitive experiments are automated, how do future researchers gain the experience that comes from doing them? Automation might undermine the very process that develops the people capable of designing it.

One blog called this an experience-pipeline problem, which seems apt. A junior researcher needs to learn the work, but the work that used to teach them no longer exists.

The Single-Metric Problem

autoresearch defines success with one number: validation bits per byte, or val_bpb. Lower is better. It is simple and easy to compare. But real-world goals rarely fit a single metric. A model may need to be fast, accurate, and safe. Compressing such a multidimensional problem into one number loses information.

This is a first version, of course. Karpathy describes it as a recipe rather than a tool: take the idea and adapt it to your own problem.

What Has the Community Done? #

The project reached 20,000 GitHub stars in four days, and forks appeared quickly. There is an MLX port for Apple Silicon, a separate Windows RTX fork, and added AMD support. A related project called AgentHub gained a thousand stars in 48 hours.

It even has entries on Ekşi Sözlük, where people are discussing Karpathy’s Software 3.0 framing. Software 1.0 is code written by humans. Software 2.0 is weights learned by neural networks. Software 3.0 is instructions written by humans in natural language—in this case, program.md. It is an interesting way to look at it.

What Is Karpathy Aiming For? #

Karpathy wrote that the next step should let agents collaborate asynchronously at a massive scale, something like SETI@home. The goal is not to imitate one PhD student, but a research community.

That made me imagine future ML research as a distributed system where thousands of agents test different hypotheses in parallel and share successful results. Their five-minute experiments could add up to millions of experiment-hours.

For now, that is a vision. But Karpathy’s track record suggests he often follows through.

If You Want to Try It #

With an NVIDIA GPU, setup is straightforward:

Terminal window
curl -LsSf https://astral.sh/uv/install.sh | sh
git clone https://github.com/karpathy/autoresearch.git
cd autoresearch
uv sync
uv run prepare.py
uv run train.py

These commands install the uv package manager, clone the repository, install dependencies, prepare the data once, and run an experiment. Data preparation takes about two minutes; an experiment takes about five.

Then tell your coding agent—Claude Code, Cursor, Copilot, or whichever you use—to read program.md and start an experiment. It handles the rest.

On a rented GPU, the cost is around $0.20 per experiment. Apple Silicon users can look at the community forks; the MLX version appears fairly stable.

For smaller GPUs or a quick trial, Karpathy recommends TinyStories. Small models in a narrower scope can still produce meaningful results.

What stayed with me is that AI is not only a race for bigger models. The more important change may be in the research process itself. Karpathy demonstrated that in 630 lines of code.

The project is open source under the MIT license. Try it and adapt it to your own field. It does not have to be machine learning: the pattern applies wherever you can try something and measure the result.

I wonder what will have changed when I return to this article in six months.