Skip to content

Pick your style: Denkbaar

Book a conversation

Articles

Knowledge

First the thinnest real line, end to end

What we learn from Karpathy's recipe for training neural networks: first build the smallest whole that truly works, and only then make it better.

A worker connects a document through one small module to a working screen.
First build the smallest real line from start to finish. Illustration: Denkbaar.

In 2019 Andrej Karpathy wrote “A Recipe for Training Neural Networks”. It is about neural networks, but the lesson fits almost everything you build, also with AI.

Why fast and rough does not work

Karpathy starts with two warnings. Training is a “leaky abstraction”: you cannot use it as a box without understanding what happens inside. And training “fails silently”: a mistake often gives no error, the result is only a bit worse. That is why, he writes, a “fast and furious” approach does not work.

An open case reveals an orange connector beside an almost normal result.
Some mistakes give no error and only make the result worse. Illustration: Denkbaar.

The recipe in six steps

  1. Become one with the data.
  2. Set up the end-to-end training and evaluation skeleton, with dumb baselines.
  3. Overfit.
  4. Regularize.
  5. Tune.
  6. Squeeze out the juice.
A developer compares sample cards with a simple chart on a laptop.
Check each improvement with a working chain and understandable measures. Illustration: Denkbaar.

Step 2 is the thinnest line

In step 2 you pick a simple model “that you couldn’t possibly have screwed up somehow”, such as a linear classifier. The goal is not a good result. The goal is a chain that truly works, from data to measurement, so that you can check every next step.

Some of his tips:

  • Always use a fixed random seed. Then the same code gives the same outcome twice.
  • Check that the loss starts at the value you expect.
  • First overfit one small batch of a few examples.
  • Also watch measures a person can read and check, such as accuracy.

How we use this

We build software the same way. First the smallest version that truly works end to end: a real call, real data, a real screen. Only then do we make it wider, stronger and nicer, and at every step we test the whole line again.

Sources

  1. Andrej Karpathy: A Recipe for Training Neural Networks (25 April 2019)

Topics: way of workingkarpathytesting