How similar are the top engines, really?

[chrisw]

How similar are the top engines, really?

Every engine developer already knows the top of the rating list looks alike — all read each other's work on GitHub. What nobody had was a metric. And no author can produce one honestly about their own code: you cannot audit your own influences from the inside. So I handed the job to an LLM that has no stake in the answer.

The ten strongest open-source engines on CCRL 40/15, plus Chess System Tal 2, compared pairwise across four languages — C, C++, Rust and Zig. Every statement is parsed to a syntax tree and canonicalised until a Rust line and a C++ line become comparable, with tuned constants, weights and variable names erased. Then the same eleven engines are scored against a 52-technique checklist, with cited evidence for each of 572 judgements.

The two answers point in opposite directions. At most 14% of search code is shared between any two engines — measured against a ceiling of 55%, which is what Stockfish scores against its own code from a year ago. Yet 37 of 52 techniques are present in every single engine, and no pair shares fewer than 78% of ideas. The strongest engines in the world have converged on one search and written it eleven times from scratch. Chess System Tal 2 is the outlier on both axes — and the only engine in the study still extending on checks.

Ten engines, one search
Pairwise code-overlap and idea-overlap matrices across eleven engines and four languages; the calibration control that makes the numbers readable; a 52-technique audit; the clean generational split over killers and countermove — half the top ten have deleted both — and where Chess System Tal 2 lands against the field.

https://whittingtonchess.com/chessenginesimilarities/

[AndrewGrant]

I would be mildly curious to produce this result over Torch's code, if I could run this myself, as there is some weird techniques in there.

But as you were arriving at, engines might start very diverse, since in the early days elo is plentiful, and novel (but bad) ideas can bring you gains. But as you mature, those things start to slip out of the picture. Internal Iterative Deepening is dead, replaced by IIR from Ed. A divide exists pre/post era on that finding. And many more such situations.

[chrisw]

I would be mildly curious to produce this result over Torch's code, if I could run this myself, as there is some weird techniques in there.

But as you were arriving at, engines might start very diverse, since in the early days elo is plentiful, and novel (but bad) ideas can bring you gains. But as you mature, those things start to slip out of the picture. Internal Iterative Deepening is dead, replaced by IIR from Ed. A divide exists pre/post era on that finding. And many more such situations.

Claude would need to be reminded of his conversation and the initial prompt. That can be written up in a handover.md for another author to use. Later tonight.

[chrisw]

Should allow for any engine, public, or private if you have the source, to be compared/contrasted as per priors (top ten engines) in this thread.

https://github.com/ChrisWhittington/che ... ch-overlap

[j.t.]

You might be interested in this: https://github.com/tsoj/engine_similarity

[AndrewGrant]

Gave it a run. Can upload some of the results. Looks like the produced rare_examples/strong_examples requires very little censoring, since the extracted stuff is quite boilerplate-like. The grammar based approach I think is quite interesting.

I'm not making any sort of attack on the methodology here, since such a problem is extremely hard in nature. But I am questioning the comparative value, just on account of the statements being tracked. But maybe I was reading too much into it. I was thinking this would identify the unique pruning methods in Torch, but it is more just basics.

Code: Select all

====================================================================================================
Stockfish / Torch  rare=11
====================================================================================================

  key: (CALL (FIELD (FIELD V0 V1) V2) (ARGS V3 (FIELD (BIN + V0 LIT) V1)))
    Stockfish    search.cpp:1491        ss->pv->update(move, (ss + 1)->pv);
    Torch        search.cpp:937         pd->pv.set_pv(best_move, (pd+1)->pv);

  key: (IF (BIN && V0 (BIN < V1 (BIN + (FIELD V2 V3) LIT))) BODY)
    Stockfish    search.cpp:764         if (PvNode && selDepth < ss->ply + 1) selDepth = ss->ply + 1;
    Torch        search.cpp:434         if (is_pv && seldepth < pd->ply + 1) seldepth = pd->ply + 1;

  key: (ASSIGN = V0 (CALL V1 V2 (ARGS (BIN - V3 V4) (UN - V5))))
    Stockfish    search.cpp:377         alpha = std::max(avg - delta, -VALUE_INFINITE);
    Torch        search.cpp:271         alpha = std::max(score - delta, -CHECKMATE);

Here is the primary matrix:

Code: Select all

=== PAIRWISE: shared distinct canonical statements ===
(pct = shared / smaller engine module)

             Stockfis pawnocch Reckless PlentyCh    Torch Obsidian Alexandr  Berserk  Halogen Quantica   Caissa
Stockfish           -       28       29       44       33       36       21       16       20       20       40
pawnocchio         28        -       14       30       20       24       17        8       17       12       22
Reckless           29       14        -       19       11       16       13       14       11       11       21
PlentyChess        44       30       19        -       28       44       32       26       25       19       29
Torch              33       20       11       28        -       29       23       23       18       13       24
Obsidian           36       24       16       44       29        -       29       24       19       16       29
Alexandria         21       17       13       32       23       29        -       29       19       24       23
Berserk            16        8       14       26       23       24       29        -       13       24       25
Halogen            20       17       11       25       18       19       19       13        -       12       29
Quanticade         20       12       11       19       13       16       24       24       12        -       14
Caissa             40       22       21       29       24       29       23       25       29       14        -

If you're a guy looking for interesting things, maybe the "PVTrees" search in Torch would interest you. A sort of multi-multi-..-multi-pv searcher with a defined tree shape limitation, which produces a tree of analysis, not just a line. No strength utility in that, but lots of practical utility.

[chrisw]

Gave it a run. Can upload some of the results. Looks like the produced rare_examples/strong_examples requires very little censoring, since the extracted stuff is quite boilerplate-like. The grammar based approach I think is quite interesting.

I'm not making any sort of attack on the methodology here, since such a problem is extremely hard in nature. But I am questioning the comparative value, just on account of the statements being tracked. But maybe I was reading too much into it. I was thinking this would identify the unique pruning methods in Torch, but it is more just basics.

Code: Select all

====================================================================================================
Stockfish / Torch  rare=11
====================================================================================================

  key: (CALL (FIELD (FIELD V0 V1) V2) (ARGS V3 (FIELD (BIN + V0 LIT) V1)))
    Stockfish    search.cpp:1491        ss->pv->update(move, (ss + 1)->pv);
    Torch        search.cpp:937         pd->pv.set_pv(best_move, (pd+1)->pv);

  key: (IF (BIN && V0 (BIN < V1 (BIN + (FIELD V2 V3) LIT))) BODY)
    Stockfish    search.cpp:764         if (PvNode && selDepth < ss->ply + 1) selDepth = ss->ply + 1;
    Torch        search.cpp:434         if (is_pv && seldepth < pd->ply + 1) seldepth = pd->ply + 1;

  key: (ASSIGN = V0 (CALL V1 V2 (ARGS (BIN - V3 V4) (UN - V5))))
    Stockfish    search.cpp:377         alpha = std::max(avg - delta, -VALUE_INFINITE);
    Torch        search.cpp:271         alpha = std::max(score - delta, -CHECKMATE);

Here is the primary matrix:

Code: Select all

=== PAIRWISE: shared distinct canonical statements ===
(pct = shared / smaller engine module)

             Stockfis pawnocch Reckless PlentyCh    Torch Obsidian Alexandr  Berserk  Halogen Quantica   Caissa
Stockfish           -       28       29       44       33       36       21       16       20       20       40
pawnocchio         28        -       14       30       20       24       17        8       17       12       22
Reckless           29       14        -       19       11       16       13       14       11       11       21
PlentyChess        44       30       19        -       28       44       32       26       25       19       29
Torch              33       20       11       28        -       29       23       23       18       13       24
Obsidian           36       24       16       44       29        -       29       24       19       16       29
Alexandria         21       17       13       32       23       29        -       29       19       24       23
Berserk            16        8       14       26       23       24       29        -       13       24       25
Halogen            20       17       11       25       18       19       19       13        -       12       29
Quanticade         20       12       11       19       13       16       24       24       12        -       14
Caissa             40       22       21       29       24       29       23       25       29       14        -

If you're a guy looking for interesting things, maybe the "PVTrees" search in Torch would interest you. A sort of multi-multi-..-multi-pv searcher with a defined tree shape limitation, which produces a tree of analysis, not just a line. No strength utility in that, but lots of practical utility.

Its quite neat how it picks a way past variable name changes and so on. But very precise, if, for example

Torch search.cpp:434 if (is_pv && seldepth < pd->ply + 1) seldepth = pd->ply + 1;
AnOther engine if ((is_pv || v >= beta) && seldepth < pd->ply + 1) seldepth = pd->ply + 1;

would not flag up as a match. Slightest change to algorithm == no match. Anyway, I took no part at all in determining the how of it, everything was decided by Claude.

[jdart]

It would be nice if this ran directly against the public repo (for those engines that have one), vs. assuming there is a local copy of the engine.

[chrisw]

It would be nice if this ran directly against the public repo (for those engines that have one), vs. assuming there is a local copy of the engine.

It does, unless I misunderstood you.

[Uri Blass]

How similar are the top engines, really?

Every engine developer already knows the top of the rating list looks alike — all read each other's work on GitHub. What nobody had was a metric. And no author can produce one honestly about their own code: you cannot audit your own influences from the inside. So I handed the job to an LLM that has no stake in the answer.

The ten strongest open-source engines on CCRL 40/15, plus Chess System Tal 2, compared pairwise across four languages — C, C++, Rust and Zig. Every statement is parsed to a syntax tree and canonicalised until a Rust line and a C++ line become comparable, with tuned constants, weights and variable names erased. Then the same eleven engines are scored against a 52-technique checklist, with cited evidence for each of 572 judgements.

The two answers point in opposite directions. At most 14% of search code is shared between any two engines — measured against a ceiling of 55%, which is what Stockfish scores against its own code from a year ago. Yet 37 of 52 techniques are present in every single engine, and no pair shares fewer than 78% of ideas. The strongest engines in the world have converged on one search and written it eleven times from scratch. Chess System Tal 2 is the outlier on both axes — and the only engine in the study still extending on checks.

Ten engines, one search
Pairwise code-overlap and idea-overlap matrices across eleven engines and four languages; the calibration control that makes the numbers readable; a 52-technique audit; the clean generational split over killers and countermove — half the top ten have deleted both — and where Chess System Tal 2 lands against the field.

https://whittingtonchess.com/chessenginesimilarities/

I see mo logical reason that they have to use one search.
I did not implement it but thought in the past about the idea of having alphabeta function that is based on probability and not on depth.

Basically if you are 100% sure that a line is bad you can prune it so you can search only lines that have probability of more than 0 to be best.
For example
First iteration you search only lines that you believe have probability of at least 0.1 to be a good line
Secomd iteration you search only lines that you believe have probability of at least 0.05 to be good and every iteration you diivide the probability by 2(you may use different factor than 2 if it gives better results).

Of course you can have also alpha and beta in the paramers and good is relative to alpha or beta(dependent on the target of the side to move) but basically I would like search with parameters alpha,beta and probability instead of depth).