Testing LazySMP

[hgm]

Well, that is not too bad, for an engine without TT. It cannot be that buggy.

The high depth observed by Ed suggests it does a lot of forward pruning.

[supernova]

Final Result: I believe this engine falls within the range of 2000 and 2100 Elo CCRL, on par with Rival 0094-ja. It has struggled against micro-Max 4.8 and has little to no chance against the Maia Lc0 networks.

As for the conditions, I will list them down here:
- Hardware Threads: 1024
- Ponder: Off
- Engines using their own books and learning enabled when possible.
- Time Setup: 2m-5sec in (Increment).
- All 3,4,5 endgames and most importantly 6-7.

LazySMP 3.0 :

Code: Select all

Acquacchi 1.0.Dev099e907-ja          :   20  :   7+  :   3=  :  10-  :   42.5%
AnMon 5.75                           :   20  :   1+  :   2=  :  17-  :   10.0%
Aramis 1.40                          :   20  :   0+  :   0=  :  20-  :    0.0%
Belofte 2.1.8.1                      :   20  :  20+  :   0=  :   0-  :  100.0%
Colchess 8.0-ja                      :   20  :  12+  :   1=  :   7-  :   62.5%
Dragon 4.6                           :   20  :   3+  :   2=  :  15-  :   20.0%
EveAnn 1.73                          :   20  :   0+  :   1=  :  19-  :    2.5%
EveAnn 2.1nn                         :   20  :   0+  :   0=  :  20-  :    0.0%
Gullydeckel-2.16.pl2                 :   20  :  10+  :   4=  :   6-  :   60.0%
Heimdall 1.1.1                       :   20  :   0+  :   0=  :  20-  :    0.0%
Hermann 2.8                          :   20  :   0+  :   0=  :  20-  :    0.0%
InmiChess 3.0.5                      :   20  :   4+  :   5=  :  11-  :   32.5%
Joker 1.1.14                         :   20  :   5+  :   1=  :  14-  :   27.5%
Lc0.cuda 0.31.1 Mean-girl.7 6t1024h  :   20  :   0+  :   0=  :  20-  :    0.0%
Lc0.cuda 0.31.1 Mean-girl.8 6t1024h  :   20  :   0+  :   0=  :  20-  :    0.0%
Lc0.cuda 0.31.1 maia-1100 6t1024h    :   20  :   2+  :   0=  :  18-  :   10.0%
Lc0.cuda 0.31.1 maia-1200 6t1024h    :   20  :   0+  :   1=  :  19-  :    2.5%
Lc0.cuda 0.31.1 maia-1300 6t1024h    :   20  :   0+  :   0=  :  20-  :    0.0%
Lc0.cuda 0.31.1 maia-1400 6t1024h    :   20  :   1+  :   0=  :  19-  :    5.0%
Lc0.cuda 0.31.1 maia-1500 6t1024h    :   20  :   0+  :   0=  :  20-  :    0.0%
Lc0.cuda 0.31.1 maia-1600 6t1024h    :   20  :   0+  :   0=  :  20-  :    0.0%
Lc0.cuda 0.31.1 maia-1700 6t1024h    :   20  :   0+  :   0=  :  20-  :    0.0%
Lc0.cuda 0.31.1 maia-1800 6t1024h    :   20  :   0+  :   0=  :  20-  :    0.0%
Lc0.cuda 0.31.1 maia-1900 6t1024h    :   20  :   0+  :   0=  :  20-  :    0.0%
MangoPaola 1.0                       :   20  :   4+  :   3=  :  13-  :   27.5%
Nalwald 19                           :   20  :   0+  :   0=  :  20-  :    0.0%
Nejmet 3.07                          :   20  :   2+  :   4=  :  14-  :   20.0%
ProDeo 1.1                           :   20  :   0+  :   2=  :  18-  :    5.0%
ProDeo 1.1 Absolute-Novice-Player    :   20  :  20+  :   0=  :   0-  :  100.0%
ProDeo 1.1 Down                      :   20  :   2+  :   0=  :  18-  :   10.0%
ProDeo 1.1 Strong-Club-Player        :   20  :   1+  :   2=  :  17-  :   10.0%
ProDeo 1.1 Style.Aggressive-AntiGM.Smart :   20  :   0+  :   1=  :  19-  :    2.5%
ProDeo 2.1 Aggressive                :   20  :   0+  :   0=  :  20-  :    0.0%
ProDeo 2.1 Average-Club-Player       :   20  :   1+  :   0=  :  19-  :    5.0%
ProDeo 2.1 Strong-Club-Player        :   20  :   0+  :   0=  :  20-  :    0.0%
Rebel Mephisto-Gideon                :   20  :   1+  :   0=  :  19-  :    5.0%
Rebel Mephisto-Gideon Creative       :   20  :   5+  :   0=  :  15-  :   25.0%
Rebel Mephisto-Gideon Style.Aggressive-AntiGM.Smart :   20  :   1+  :   0=  :  19-  :    5.0%
Rival 0094-ja                        :   20  :   9+  :   2=  :   9-  :   50.0%
RoboKewlper 0.047a                   :   20  :  20+  :   0=  :   0-  :  100.0%
Skaki 1.23-ja                        :   20  :  18+  :   2=  :   0-  :   95.0%
StAndersen 1.31c-rev2-ja             :   20  :  19+  :   1=  :   0-  :   97.5%
Tcheran 4.0                          :   20  :   0+  :   1=  :  19-  :    2.5%
The King 3.50                        :   20  :   0+  :   0=  :  20-  :    0.0%
Toad 1.0                             :   20  :  11+  :   5=  :   4-  :   67.5%
Tscp 181e-32-ja                      :   20  :  17+  :   1=  :   2-  :   87.5%
Yace 0.99.87                         :   20  :   2+  :   2=  :  16-  :   15.0%
micro-Max 4.8                        :   20  :  12+  :   3=  :   5-  :   67.5%

https://drive.google.com/file/d/1aPIFcY ... sp=sharing

Sorry, it was Hash Table: 1024 MB and not [Hardware Threads: 1024]

[LazySMP]

Well, that is not too bad, for an engine without TT. It cannot be that buggy.

The high depth observed by Ed suggests it does a lot of forward pruning.

Thanks for your support and I'm glad you were reading. The LazySMP version 3.0 was a PST/material only version. Surely, if I had improved evaluation function, it would have been much stronger. If you are interested, I would like to develop a new chess engine with your collaboration.

Of course, there may be many people against it and it will never be tested by CCRL members

There are two kinds of perfts: where you just generate the legal moves and determine how many you generated from the length of the list, and where you also make and unmake all these moves. The former basically measures the speed of your move generation, the latter the speed of your make/unmake. Typically move generation takes much more time than making/unmaking a single move.

There is no need to do / undo the move at the last ply. https://github.com/mcostalba/Stockfish/ ... 58afb1b64a

[hgm]

Bad idea. I am not good at cooperating, not really interested in orthodox Chess, and are currenly so busy with things outside chess programming that I even had to put the development of my latest project on hold for indeterminate time.

And indeed, making/unmaking the final ply is not needed, but then again perft itself is not needed. It is just an exercise for measuring speed of certain components of a chess engine. And to interpret the measurement it is essential to know what has been measured.

The point was that typical engines end the branches of their search tree with move generation. Not with making/unmaking moves. So perfts that count the number of make/unmakes rather than the number of move generations give little or no insight in the speed of an engine using that same generation/make algorithm. The large discrepancy between the mentioned 35Mnps and 2Mnps already testify to that.

[Ras]

The LazySMP version 3.0 was a PST/material only version. Surely, if I had improved evaluation function, it would have been much stronger.

Did you already get around to machine tuned PST and material values?

There is no need to do / undo the move at the last ply.

That depends on how the engine implements the legality check. If that works with make move / in-check detection / unmake move, then there has to be make/unmake in the last ply. Otherwise, like in Stockfish, the number of available legal moves can be counted, which is called "bulk counting". Whatever method is used in perft should also mirror what the engine is doing in gameplay so that perft can be used to verify the correctness, especially in weird test positions.

[hgm]

But an engine, as opposed to perft, has no reason to do a legality check on moves that it does not want to search.

[LazySMP]

Bad idea. I am not good at cooperating, not really interested in orthodox Chess, and are currenly so busy with things outside chess programming that I even had to put the development of my latest project on hold for indeterminate time.

I apologize for my bad idea. I thought the CCRL member criteria for testing engines was the author's name, as you said they tested your engine "Joker" when it wasn't open source. While they tested private engines like "Torch" just because of the author's name, I'm not sure if this engine is a copy of other engines or not!

That depends on how the engine implements the legality check. If that works with make move / in-check detection / unmake move, then there has to be make/unmake in the last ply. Otherwise, like in Stockfish, the number of available legal moves can be counted, which is called "bulk counting". Whatever method is used in perft should also mirror what the engine is doing in gameplay so that perft can be used to verify the correctness, especially in weird test positions.

In some engines, usually old ones, it is necessary to make move to check legal king moves. So, there is need to do/undo the move at the last ply.

Code: Select all

    for (var i = 0, len = moves.length; i < len; i++) {
        make_move(moves[i])
        if (!king_attacked(color)) {
            var child_nodes = perft(depth - 1)
            nodes += child_nodes
        }
        undo_move()
    }

[Ras]

But an engine, as opposed to perft, has no reason to do a legality check on moves that it does not want to search.

That's correct, but it does need to check all moves that it wants to search - which, in case of perft, happen top be all pre-leaf moves.

In some engines, usually old ones, it is necessary to make move to check legal king moves. So, there is need to do/undo the move at the last ply.

Yes, that's the point. Though I changed to a hybrid approach where I can mostly omit the post-move in-check verification. Using the alignment of moving piece and king to rule out the necessity for post-move verification allows to completely omit the make / verify / unmake chain for a lot of leaf nodes in perft, which is where the jump from 35MNPS to 80MPNS in perft came from. Though that was only a side effect, the actual objective was the speed-up in gameplay, which obviously was much less than doubling (something like 10% or so).

[hgm]

But an engine, as opposed to perft, has no reason to do a legality check on moves that it does not want to search.

That's correct, but it does need to check all moves that it wants to search - which, in case of perft, happen top be all pre-leaf moves.

I am not sure what point you want to make with this. If an engine makes a (legal) move to search, this will always be followed by a move generation. For the moves you 'search' in the last ply of perft making them is never followed by a move generation. So they are very different. A perft with bulk counting is much more similar to an engine search: the branches there also end in move generation. Especially if your move generator already refrains from generating moves of pinned pieces that would expose your King. Then you would only have to test King moves for legality, of which there aren't very many. And which you could also do without making them.

[hgm]

I thought the CCRL member criteria for testing engines was the author's name, as you said they tested your engine "Joker" when it wasn't open source. While they tested private engines like "Torch" just because of the author's name, I'm not sure if this engine is a copy of other engines or not!

Normally they would test private engines if there is not a strong suspicion that these are copies or close derivatives of other engines. Such suspicion can arise from high similarity in move choice or other behaviors, or from the simple fact that people who never programmed an engine before pop up out of nowhere with an engine stronger than 2600 Elo. But trusting someone always entails the risk of being fooled, such as for instance happened in the case of Houdini. For an author that has a history of creating and continually improving chess engines, some of those open source, there would be little reason to believe he is cheating.

In some engines, usually old ones, it is necessary to make move to check legal king moves. So, there is need to do/undo the move at the last ply.

Code: Select all

    for (var i = 0, len = moves.length; i < len; i++) {
        make_move(moves[i])
        if (!king_attacked(color)) {
            var child_nodes = perft(depth - 1)
            nodes += child_nodes
        }
        undo_move()
    }

This is code from perft, not from an engine.