Kai Chung

July 29, 2026 · Chess Engines

Zero-Bootstrapping a NNUE from Scratch

When building my chess engine Onslaught, our ƎUИИ (NNUE, Efficiently Updatable Neural Network) began as a small network trained on a handtuned classical evaluation (HCE). One topic that fascinated me was removing human bias from evaluation.

Note: early (before generation 4) Elo figures below are rough estimates, not SPRT-verified

Bootstrapping NNUE

NNUE is a pure-value network. That is, it will take in a chess position by accumulator (its feature encoding), and produce an integer without policy. Thus, NNUE needs data that consists of chess positions and at least one evaluation metric. Normally, NNUE is trained on a combination of a previous evaluation score and a game outcome. As mentioned previously, we started by using our search evaluations while utilizing HCE.

Zero-bootstrapping

Zero-Bootstrapping is a different way of seeding NNUE. The way it works is instead of starting with training data from HCE + search as our seed for a network, we start with a randomly initialized network. This first network, when plugged into search will make nonsensical moves, as expected. However, the search helps prevent obvious checkmates, finds basic tactics, and builds a "ground truth" of the game. Then, we trained another network on data from this random network. Here, we can't rely at all on the produced evaluations from the network, so the next network is completely trained on game outcome. Then, we rinse and repeat with the resulting networks, while training on the evaluation value increasingly over time, as the value becomes more and more trustworthy.

Initial architecture

Up until later generations of the network, we started with a very basic NNUE architecture. The board is encoded into 768 inputs (64 squares x 12 pieces) and passed through a dense layer. Our dense layer was relatively large, with a size of 1024 neurons. The benefits of this was primarily simplicity of implementation (as we already had this architecture implemented), and it takes less data to generalize the network.

Initial progress

Our initial trained network (generation 1) was leaps and bounds better than the random network. It played weak chess, but definitely recognizable chess. We never tested its strength, because we were primarily interested in creating the next generation, but I estimated it, with search, to play about an 1000 Elo level. By generation 3, we had a network that played around a 2000 level. By this point, we were training on a blend of game outcome and evaluation. Interestingly, something I noticed was that generation 3 played some interesting responses to common opening lines, such as 1. e4 c5 2. Nf3 d6 3. b4?!, which is a sideline that at the engine level is a bit better for black.

A brick wall

We trained generation 4, and saw some improvements in play. At this point, we needed to verify network changes with SPRT (sequential probability ratio test, basically to determine if a change improves the engine, if it keeps it about the same strength-wise, or weakens the engine). However, we saw that generation 4 had only made a minor improvement over generation 3 (about 10-15 Elo). This was strange, as we had saw leaps of up to 1000 Elo between generations. This meant either that our training pipeline was broken, or that our network was reaching the limits of our architecture. So, naturally I tested it against our HCE trained release network, which used a similar architecture apart from a smaller hidden layer. To my delight, I found that the zero-bootstrapped network was over 50 Elo (+- 12 Elo) stronger! This meant that we had most likely almost reached the maximum capability of our simple network architecture. Notably, we had reached this point very quickly which was a surprise.

A new architecture - HalfKAv2_hm

In most modern engines, pieces are encoded relative to their king. This greatly improves endgame strength in particular, and offers strength gains all around. The initial architecture that provided this was HalfKP (Half King-Piece). We decided to skip this, and move right on to HalfKAv2_hm (Half King-All horizontally mirrored, which is the modern standard), where in addition of encoding each feature as a tuple of (our king square, piece square, piece type, piece color), we include the king itself as a feature. Kings are placed into buckets that affect the dense layer used to compute the resulting value.

The fifth generation

I trained the 5th generation network on 1.5 billion positions. 1 billion were from generation 4 and 500 million of them were from generation 3. We re-ran SPRT against both the previous network and against the release network. We found that it had drastically improved - gaining over 200 Elo over the 4th generation network.

Scheduling outcome

A couple of days ago, I trained the 6th generation network off of 2 billion positions from generation 5. I doubled the number of king buckets. Notably, instead of using a static 50% outcome and 50% score, I switched it to a linear scheduler, where it started as pure evaluation, and slowly progressed to 75% evaluation and 25% outcome. SPRT claimed that this network had gained nearly 50 additional Elo points, which is very significant for the point at we are network-wise.

Training schedule

Here's the timeline of our networks.

Takeaways

Through this process, which started as a way to remove human bias from our evaluation, has taught me a lot the value of testing and evaluation and knowing when to scale. Without testing, we would have never been able to realize that we needed a more sophisticated architecture. In the end, we reduced the bias in our engine while improving its strength, which is the definition of a win-win in my book.

Back to blog