Comparison check on engine performance

[Chessnut1071]

My mediocre and rather slow 2500-ish engine, Myrddin, needs 3.6 seconds to announce Mate in 8, and it needs ~9.5M nodes to do it. So about 2.6KNPS.

The question I have is what is in your evaluation?

[emadsen]

MadChess finds it in about 1.1 seconds on my Ryzen 1950X 3.4 GHz.

Code: Select all

position fen 1rr3k1/2q4p/2p2p2/1bQpNp2/p2P4/P3P3/5P1P/R1R4K w - - 0 1

debug on

go mate 8
info depth 1 seldepth 6 time 87 nodes 161 score cp -21 nps 1851 pv c1g1
info hashfull 0 currmove c5a7 currmovenumber 37
info string Cache Hit = 0.00% Score Cutoff = NaN% Best Move Hit = 0.00% Invalid Best Moves = 0
info string Null Move Cutoffs = NaN% Beta Cutoff Move Number = NaN Beta Cutoff First Move = NaN%
info string Evals = 144
info depth 2 seldepth 6 time 112 nodes 247 score cp -21 nps 2201 pv c1g1 g8h8
info hashfull 0 currmove c5a7 currmovenumber 37
info string Cache Hit = 1.16% Score Cutoff = 0.00% Best Move Hit = 11.11% Invalid Best Moves = 0
info string Null Move Cutoffs = NaN% Beta Cutoff Move Number = 1.00 Beta Cutoff First Move = 100.00%
info string Evals = 222
info depth 3 seldepth 8 time 119 nodes 609 score cp -57 nps 5115 pv c1g1 g8h8 e5f3
info hashfull 0 currmove c5a7 currmovenumber 37
info string Cache Hit = 4.55% Score Cutoff = 0.00% Best Move Hit = 29.63% Invalid Best Moves = 0
info string Null Move Cutoffs = 100.00% Beta Cutoff Move Number = 1.00 Beta Cutoff First Move = 100.00%
info string Evals = 548
info depth 4 seldepth 11 time 124 nodes 1583 score cp -164 nps 12809 pv e5f3 g8h8 h2h4 h7h6
info hashfull 0 currmove c5a7 currmovenumber 37
info string Cache Hit = 5.79% Score Cutoff = 3.23% Best Move Hit = 20.37% Invalid Best Moves = 0
info string Null Move Cutoffs = 100.00% Beta Cutoff Move Number = 1.65 Beta Cutoff First Move = 82.43%
info string Evals = 1,435
info depth 5 seldepth 11 time 127 nodes 4312 score cp -229 nps 33846 pv c1g1 g8h8 e5f3 e2d3 f2f4
info hashfull 0 currmove h2h4 currmovenumber 37
info string Cache Hit = 9.77% Score Cutoff = 11.88% Best Move Hit = 24.59% Invalid Best Moves = 0
info string Null Move Cutoffs = 75.00% Beta Cutoff Move Number = 1.58 Beta Cutoff First Move = 81.86%
info string Evals = 3,858
info depth 6 seldepth 12 time 136 nodes 10892 score cp -172 nps 79923 pv e5f3 g8h8 f3h4 c7d7 c1g1 c8g8
info hashfull 0 currmove h2h4 currmovenumber 37
info string Cache Hit = 15.44% Score Cutoff = 20.16% Best Move Hit = 30.50% Invalid Best Moves = 0
info string Null Move Cutoffs = 60.98% Beta Cutoff Move Number = 1.44 Beta Cutoff First Move = 81.97%
info string Evals = 9,624
info depth 7 seldepth 17 time 148 nodes 19473 score cp 713 nps 131788 pv c1g1 g8h8 c5d6 h7h6 d6f6 h8h7 f6g6
info hashfull 0 currmove a1b1 currmovenumber 37
info string Cache Hit = 17.23% Score Cutoff = 25.38% Best Move Hit = 32.76% Invalid Best Moves = 0
info string Null Move Cutoffs = 57.39% Beta Cutoff Move Number = 1.50 Beta Cutoff First Move = 82.06%
info string Evals = 16,630
info depth 8 seldepth 15 time 150 nodes 20526 score cp 713 nps 136829 pv c1g1 g8h8 c5d6 h7h6 d6f6 h8h7 f6g6 h7h8
info hashfull 0 currmove h2h3 currmovenumber 37
info string Cache Hit = 18.72% Score Cutoff = 24.00% Best Move Hit = 33.05% Invalid Best Moves = 0
info string Null Move Cutoffs = 57.60% Beta Cutoff Move Number = 1.46 Beta Cutoff First Move = 83.26%
info string Evals = 17,452
info depth 9 seldepth 17 time 154 nodes 22900 score cp 1916 nps 148885 pv c1g1 g8h8 c5d6 b8b7 d6f6 c7g7 g1g7 b7g7 f6f5
info hashfull 0 currmove h2h3 currmovenumber 37
info string Cache Hit = 21.73% Score Cutoff = 20.23% Best Move Hit = 34.50% Invalid Best Moves = 0
info string Null Move Cutoffs = 60.38% Beta Cutoff Move Number = 1.44 Beta Cutoff First Move = 84.22%
info string Evals = 19,177
info depth 10 seldepth 16 time 156 nodes 24734 score cp 1865 nps 158288 pv c1g1 g8h8 c5d6 b8b7 d6f6 c7g7 g1g7 b7g7 f6f5 c8e8
info hashfull 0 currmove h2h3 currmovenumber 37
info string Cache Hit = 23.75% Score Cutoff = 18.63% Best Move Hit = 35.05% Invalid Best Moves = 0
info string Null Move Cutoffs = 65.44% Beta Cutoff Move Number = 1.41 Beta Cutoff First Move = 85.50%
info string Evals = 20,622
info depth 11 seldepth 16 time 160 nodes 27512 score cp 1934 nps 172040 pv c1g1 g8h8 c5d6 b8b7 d6f6 c7g7 g1g7 b7g7 f6f5 c8e8 a1g1
info hashfull 0 currmove h2h3 currmovenumber 37
info string Cache Hit = 27.99% Score Cutoff = 16.06% Best Move Hit = 36.53% Invalid Best Moves = 0
info string Null Move Cutoffs = 73.52% Beta Cutoff Move Number = 1.39 Beta Cutoff First Move = 86.34%
info string Evals = 22,730
info depth 12 seldepth 17 time 166 nodes 32481 score cp 2025 nps 195533 pv c1g1 g8h8 c5d6 b8b7 d6f6 c7g7 g1g7 b7g7 f6f5 c8e8 a1g1 g7a7
info hashfull 0 currmove h2h3 currmovenumber 37
info string Cache Hit = 30.86% Score Cutoff = 14.50% Best Move Hit = 37.25% Invalid Best Moves = 0
info string Null Move Cutoffs = 79.45% Beta Cutoff Move Number = 1.34 Beta Cutoff First Move = 87.45%
info string Evals = 26,605
info depth 13 seldepth 19 time 182 nodes 44204 score cp 2424 nps 243379 pv c1g1 g8h8 c5d6 b8b7 d6f6 c7g7 g1g7 b7g7 f6f5 c8e8 a1g1 e8g8 g1g7
info hashfull 0 currmove h2h3 currmovenumber 37
info string Cache Hit = 30.47% Score Cutoff = 14.02% Best Move Hit = 31.99% Invalid Best Moves = 0
info string Null Move Cutoffs = 81.29% Beta Cutoff Move Number = 1.38 Beta Cutoff First Move = 86.54%
info string Evals = 35,293
info depth 14 seldepth 20 time 194 nodes 54365 score cp 2395 nps 279886 pv c1g1 g8h8 c5d6 b8b7 d6f6 c7g7 g1g7 b7g7 f6f5 c8e8 a1g1 g7g1 h1g1 e8g8
info hashfull 0 currmove h2h3 currmovenumber 37
info string Cache Hit = 34.17% Score Cutoff = 14.19% Best Move Hit = 33.72% Invalid Best Moves = 0
info string Null Move Cutoffs = 83.80% Beta Cutoff Move Number = 1.34 Beta Cutoff First Move = 88.40%
info string Evals = 42,717
info depth 15 seldepth 25 time 517 nodes 382199 score cp 7259 nps 738994 pv c1g1 g8h8 c5d6 c8g8 d6c7 g8g4 e5f7 h8g8 f7h6 g8h8 c7b8 h8g7 h6f5 g7f7 g1g4
info hashfull 3 currmove h2h3 currmovenumber 37
info string Cache Hit = 30.81% Score Cutoff = 28.29% Best Move Hit = 33.87% Invalid Best Moves = 0
info string Null Move Cutoffs = 92.76% Beta Cutoff Move Number = 1.35 Beta Cutoff First Move = 90.39%
info string Evals = 299,107
info depth 16 seldepth 26 time 552 nodes 415242 score cp 7060 nps 752133 pv c1g1 g8h8 c5d6 c8g8 d6c7 g8g4 e5f7 h8g8 f7h6 g8h8 c7b8 h8g7 h6f5 g7f7 g1g4 b5e2
info hashfull 4 currmove c5b4 currmovenumber 37
info string Cache Hit = 32.40% Score Cutoff = 27.46% Best Move Hit = 34.39% Invalid Best Moves = 0
info string Null Move Cutoffs = 92.80% Beta Cutoff Move Number = 1.32 Beta Cutoff First Move = 90.87%
info string Evals = 321,770
info depth 17 seldepth 25 time 592 nodes 452633 score cp 7274 nps 764494 pv c1g1 g8h8 c5d6 c8g8 d6c7 g8g4 e5f7 h8g8 f7h6 g8h8 c7b8 h8g7 h6f5 g7f7 b8c7 f7e6 c7c8
info hashfull 5 currmove c5b4 currmovenumber 37
info string Cache Hit = 34.98% Score Cutoff = 26.72% Best Move Hit = 35.74% Invalid Best Moves = 0
info string Null Move Cutoffs = 93.06% Beta Cutoff Move Number = 1.32 Beta Cutoff First Move = 91.14%
info string Evals = 345,160
info depth 18 seldepth 25 time 644 nodes 503964 score cp 7274 nps 782221 pv c1g1 g8h8 c5d6 c8g8 d6c7 g8g4 e5f7 h8g8 f7h6 g8h8 c7b8 h8g7 h6f5 g7f7 b8c7 f7e6 c7c8 e6f7
info hashfull 6 currmove c5b4 currmovenumber 37
info string Cache Hit = 36.85% Score Cutoff = 27.03% Best Move Hit = 35.57% Invalid Best Moves = 0
info string Null Move Cutoffs = 93.32% Beta Cutoff Move Number = 1.29 Beta Cutoff First Move = 91.70%
info string Evals = 378,073
info depth 19 seldepth 27 time 1022 nodes 1000000 nps 978729
info hashfull 9 currmove c5c6 currmovenumber 5
info string Cache Hit = 32.59% Score Cutoff = 19.41% Best Move Hit = 35.73% Invalid Best Moves = 0
info string Null Move Cutoffs = 96.87% Beta Cutoff Move Number = 1.18 Beta Cutoff First Move = 94.55%
info string Evals = 776,812
info depth 19 seldepth 27 time 1132 nodes 1136051 score mate 8 nps 1004015 pv c1g1 g8h8 c5d6 c8g8 d6c7 g8g4 e5f7 h8g8 f7h6 g8h8 g1g4 f5g4 c7f7 b8g8 f7g8
info hashfull 10 currmove c5b4 currmovenumber 37
info string Cache Hit = 32.08% Score Cutoff = 18.10% Best Move Hit = 35.60% Invalid Best Moves = 0
info string Null Move Cutoffs = 97.19% Beta Cutoff Move Number = 1.17 Beta Cutoff First Move = 94.87%
info string Evals = 888,401
info string Stopping search at 1132 milliseconds.
bestmove c1g1

I assume you mean nodes per second. I'm using engine calls with approximately 40 nodes each. Using your metric 40 x 131.200,30`5,242,520 nodes per second. Also, I'm not sure what perft is. Looking for a description.

Bill, this is incorrect. There's no need to estimate nodes. Increment a running counter in your Board.PlayMove method. "Approximately 40 nodes each" makes no sense.

Instead of wasting time on move generation, Good move ordering, Transposition tables, Null Move Ordering, Futility Pruning, Reverse Futility Pruning, Late Move Pruning, Late Move Reduction, and most importantly very good evaluation would be much more helpful to your engine strength.

+1

[lithander]

OMG, it took me 2 months to get the bitboards working after correcting all the known issues. This looks like a never ending task. I didn't think 5.2Mnps was any good. I'm reading some clocking 120Mnps. Are those speeds on a personal computer?

You need to be very careful what you compare. I'm currently focusing on writing a speedy engine in C# and my perft was running at 69Mnps. Thats quite fast for a pseudo-legal move generator where each visited node needs to be validated for legality, too. But as soon as you add features you'll see the nodes per second drop further and further. So 5M nps can be good or bad speed; It depends entirely on what you already do besides just generating and playing moves.

My new engine is 6x faster then my old engine but it hasn't all the features yet. The branching factor is higher and so finding a mate in 8 takes forever even on 6M nps whereas my old, slow engine (less than 1M nps) finds the mate in 8 in less than 4 seconds.

Better search&eval will yield exponential gains while raw speed will just yield linear gains. Still... if I have implemented the same features in a 6M nps engine and a 1M nps engine I can, for example, use faster time controls in self-testing to validate an improvement with the same statistical accuracy. My auto-tuner will complete much faster. And so on... a linear increase of even 2 or 3 in developing speed is a game changer for something that is a hobby. So *that* was my main reason to rewrite my old engine and focus on speed the 2nd time.

[lithander]

My mediocre and rather slow 2500-ish engine, Myrddin, needs 3.6 seconds to announce Mate in 8, and it needs ~9.5M nodes to do it. So about 2.6KNPS.

2.6M NPS you mean, right?

[mvanthoor]

OMG, it took me 2 months to get the bitboards working after correcting all the known issues. This looks like a never ending task. I didn't think 5.2Mnps was any good. I'm reading some clocking 120Mnps. Are those speeds on a personal computer?

What are you comparing? In my case, Perft runs at 33M nps on an i7-6700K... but the engine:
- Incrementally keeps a Zobrist Hash
- Incrementally keeps 2 sets of PSQT's
- Incrementally keeps the game phase

The two sets of PSQT's and the game phase cost about 8M nps to keep. I never measured the Zobrist key but I expect it to take another 4M. So the engine could have run at ~47M nps. (In the past it did run at 41M nps when it didn't have the PSQT's and phase yet, but it did have the Zobrist hash.)

All of this could be scrapped for Perft. Also, I could add bulk-counting to perft, and change the move generator into a full-legal move generator to increase speed even more. However, all of this has no bearing on chess playing strength. It's just Perft.

The engine searches 6 M nodes/second when running alpha/beta and evaluating positions. THAT is what counts, because there, I DO need the incremental zobrist hash, the PSQT's, and in my case, the game phase.

So if you are running at 5.2 M nps with your engine while searching positions, that is completely fine. That should be enough to build a seriously strong engine on top of. (Except if you are reaching those 5.2 M nodes/second on a supercomputer, obvioulsy. I'm talking about reaching this on a 5 year old laptop.)

[mvanthoor]

Better search&eval will yield exponential gains while raw speed will just yield linear gains. Still... if I have implemented the same features in a 6M nps engine and a 1M nps engine I can, for example, use faster time controls in self-testing to validate an improvement with the same statistical accuracy. My auto-tuner will complete much faster. And so on... a linear increase of even 2 or 3 in developing speed is a game changer for something that is a hobby. So *that* was my main reason to rewrite my old engine and focus on speed the 2nd time.

What I also said in the past is: Speed is currency (money) in chess engines.

Making your evaluation bigger costs speed. You have to be careful to only implement things that gain MORE in evaluation terms than they cost in speed. However, if you have an engine that runs at 6M nodes/second and another that runs at 1M nodes/second (with exactly the same feature set), you can implement a HUGE amount of evaluation features in that faster engine before it has dropped down to 1M nodes/second; then the engines will be the same speed when measured in nodes/second, but the faster engine will have lots more evaluation features.

So with speed you can buy more features.

(This is what Hiarcs did in the past. It couldn't search as deep as other engines because it had a HUGE evaluation, but because of its better positional awareness, it could keep up with engines like Shredder and Fritz. The upshot of Hiarcs' evaluation was that it was said to play much better chess than the other engines: more human-like, less computer-like.)

[pedrojdm2021]

My primitive and slow engine only can get up to depth 13 in 3.2 seconds (in my old AMD FX 8350 CPU, single core processing, i don't use multi-threading). It searches 4356671 nodes
with the following pv:

Code: Select all

pv c1g1 g8h8 c5d6 c7g7 g1g7 c8f8 g7f7 f8f7 d6b8 h8g7 a1g1 g7h6 e5f7 h6h5 b8d6 h5h4 d6f6

more than that, is going to take hours i think.

[JVMerlino]

My mediocre and rather slow 2500-ish engine, Myrddin, needs 3.6 seconds to announce Mate in 8, and it needs ~9.5M nodes to do it. So about 2.6KNPS.

2.6M NPS you mean, right?

yep....

[JVMerlino]

My mediocre and rather slow 2500-ish engine, Myrddin, needs 3.6 seconds to announce Mate in 8, and it needs ~9.5M nodes to do it. So about 2.6MNPS.

The question I have is what is in your evaluation?

All the basic stuff (material, PST, pawn structure, mobility, simple king safety, bishop pair, bishop/knight outposts, rooks on open files), tuned with a VERY rudimentary Texel tuning implementation, with a standard tapered eval.

[brianr]

Methods: alpha/beta, Zobirst Hash
Evaluation Priority:
1-double check 2: discovered check 3: single check
4: capture piece type 1-P 2-N 3-B 4-R 5-Q
5: History
6: Pawn Promotion
7: number of enemy king moves
8: ent passant
Data capture per move:
1 piece 2 move 3: capture 4: direction 5: check type 6: from 7: to 8: check direction from e king
9: skip bit 10:Pawn promotion 11: ent passant 12: 2-step pawn move

I'm not clear about your Evaluation Priority (EP) v Methods.
If Methods refers to search and EP is actually in your evaluation there is a major difference from many engines, including my ancient and pretty meh engine Tinker, which finds the mate solution in .34 sec and is only 1.2Mnps from the starting position (still only single thread search).

There are two main concerns.
Firstly, do you use null move? Null move is the single most important thing to add to improve search performance (although it can be tricky to get working).
Secondly, I don't do anything with being in check in the evaluation, as being in check is not considered a quiescent position, so q-search is not called when in check, which in turn means the eval is not called (other than for search pruning cases, IIRC). Testing being in check and doing move counts is very expensive, especially at the evaluation level (leaves) of the tree. Simply extend the main search whenever in check (with a limit to avoid runaway positions). Like null move for pruning, the extend when in check extension is by far the most important.

FWIW Tinker reports three depth values: current iteration (of IID), extended search depth, and deepest depth in q-search.
For the mate problem it looks like this:
tellics kibitz +299.81 at 13(19/24) ply 0.34 secs 339 Knps pv: 1. c1g1 g8h8 2. c5d6 c8g8 3. d6c7 g8g4