Knowledge
First the thinnest real line, end to end
What we learn from Karpathy's recipe for training neural networks: first build the smallest whole that truly works, and only then make it better.

In 2019 Andrej Karpathy wrote “A Recipe for Training Neural Networks”. It is about neural networks, but the lesson fits almost everything you build, also with AI.
Why fast and rough does not work
Karpathy starts with two warnings. Training is a “leaky abstraction”: you cannot use it as a box without understanding what happens inside. And training “fails silently”: a mistake often gives no error, the result is only a bit worse. That is why, he writes, a “fast and furious” approach does not work.

The recipe in six steps
- Become one with the data.
- Set up the end-to-end training and evaluation skeleton, with dumb baselines.
- Overfit.
- Regularize.
- Tune.
- Squeeze out the juice.

Step 2 is the thinnest line
In step 2 you pick a simple model “that you couldn’t possibly have screwed up somehow”, such as a linear classifier. The goal is not a good result. The goal is a chain that truly works, from data to measurement, so that you can check every next step.
Some of his tips:
- Always use a fixed random seed. Then the same code gives the same outcome twice.
- Check that the loss starts at the value you expect.
- First overfit one small batch of a few examples.
- Also watch measures a person can read and check, such as accuracy.
How we use this
We build software the same way. First the smallest version that truly works end to end: a real call, real data, a real screen. Only then do we make it wider, stronger and nicer, and at every step we test the whole line again.