Stockfish benchmark

[Jouni]

Because it's not reliable I took a closer look, why? First I must notice, that I get 10% BETTER score, if I print output to file with > log.txt command than when printing to display (windows7)! With "stockfish.exe bench 256 2 20 default depth" I get with 2.2.2 1,86 Mb/s to display and 1,93 Mb/s to file. In benchmark.cpp I see it measures total time and nodes and calculates ratio. But I think it is better to count average NPS for these 16 positions. E.g. from log.txt

info depth 20 seldepth 26 score cp 36 nodes 10434518 nps 1760208
info depth 20 seldepth 32 score cp -181 nodes 26116595 nps 1935422
info depth 20 seldepth 44 score cp 113 nodes 3685658 nps 2953251
info depth 20 seldepth 30 score cp 72 nodes 15022894 nps 2159392
info depth 20 seldepth 32 score cp 202 nodes 3249327 nps 1964526
info depth 20 seldepth 24 score cp 0 nodes 4490634 nps 1881287
info depth 20 seldepth 29 score cp -56 nodes 11995418 nps 1982058
info depth 20 seldepth 30 score cp 254 nodes 22919343 nps 2086801
info depth 20 seldepth 31 score cp 52 nodes 17469128 nps 1832874
info depth 20 seldepth 36 score cp 113 nodes 10576686 nps 2158947
info depth 20 seldepth 30 score cp -145 nodes 15670760 nps 1816478
info depth 20 seldepth 29 score cp 64 nodes 77144230 nps 1757351
info depth 20 seldepth 32 score cp 210 nodes 15178871 nps 1926741
info depth 20 seldepth 31 score cp -20 nodes 26025658 nps 1849595
info depth 20 seldepth 28 score cp 121 nodes 2742549 nps 2283554
info depth 20 seldepth 39 score cp 141 nodes 6821552 nps 2196957

This gives real speed of 2,033 kn/s!

Jouni

[zullil]

Because it's not reliable I took a closer look, why? First I must notice, that I get 10% BETTER score, if I print output to file with > log.txt command than when printing to display (windows7)! With "stockfish.exe bench 256 2 20 default depth" I get with 2.2.2 1,86 Mb/s to display and 1,93 Mb/s to file. In benchmark.cpp I see it measures total time and nodes and calculates ratio. But I think it is better to count average NPS for these 16 positions. E.g. from log.txt

info depth 20 seldepth 26 score cp 36 nodes 10434518 nps 1760208
info depth 20 seldepth 32 score cp -181 nodes 26116595 nps 1935422
info depth 20 seldepth 44 score cp 113 nodes 3685658 nps 2953251
info depth 20 seldepth 30 score cp 72 nodes 15022894 nps 2159392
info depth 20 seldepth 32 score cp 202 nodes 3249327 nps 1964526
info depth 20 seldepth 24 score cp 0 nodes 4490634 nps 1881287
info depth 20 seldepth 29 score cp -56 nodes 11995418 nps 1982058
info depth 20 seldepth 30 score cp 254 nodes 22919343 nps 2086801
info depth 20 seldepth 31 score cp 52 nodes 17469128 nps 1832874
info depth 20 seldepth 36 score cp 113 nodes 10576686 nps 2158947
info depth 20 seldepth 30 score cp -145 nodes 15670760 nps 1816478
info depth 20 seldepth 29 score cp 64 nodes 77144230 nps 1757351
info depth 20 seldepth 32 score cp 210 nodes 15178871 nps 1926741
info depth 20 seldepth 31 score cp -20 nodes 26025658 nps 1849595
info depth 20 seldepth 28 score cp 121 nodes 2742549 nps 2283554
info depth 20 seldepth 39 score cp 141 nodes 6821552 nps 2196957

This gives real speed of 2,033 kn/s!

Jouni

Let's eliminate non-determinism. Just use one thread, i.e, stick to

Code: Select all

stockfish.exe bench

You should get

Code: Select all

Nodes searched  : 5447426

each time, so forget the "nodes/second" number and focus on the "Total time (ms) number"

Do you really get significantly different values for this number based on whether or not you redirect the output to a file? I don't, using my own compile of Stockfish-2.2.2 on Mac OS X.

For example, without redirection I got 4379, 4382, 4375, 4388, 4378.

With redirection, I got 4381, 4382, 4382, 4378, 4380.

The sums are 21902 and 21903 respectively.

[Jouni]

Because there is 16 positions and 20 plies non-determinism is minimal. But here's bench with 1 cpu and 16 ply:

to display 2 tries:

===========================
Total time (ms) : 51043
Nodes searched : 48381913
Nodes/second : 947865

===========================
Total time (ms) : 51309
Nodes searched : 48381913
Nodes/second : 942951

to file 2 tries:

===========================
Total time (ms) : 48345
Nodes searched : 48381913
Nodes/second : 1000763

===========================
Total time (ms) : 48376
Nodes searched : 48381913
Nodes/second : 1000122

So is displaying is slower than filewriting

Jouni

[zullil]

Here's what I get:

Without redirection to file:

Code: Select all

===========================
Total time (ms) : 40289
Nodes searched  : 48381913
Nodes/second    : 1200871


===========================
Total time (ms) : 40293
Nodes searched  : 48381913
Nodes/second    : 1200752

With redirection to file:

Code: Select all

===========================
Total time (ms) : 40454
Nodes searched  : 48381913
Nodes/second    : 1195973


===========================
Total time (ms) : 40434
Nodes searched  : 48381913
Nodes/second    : 1196565

I used ./stockfish-2.2.2 bench 256 1 16 default depth

Something related to timing seems wrong with your Windows binary. I guess you've been saying that for a while.

[bob]

Here's what I get:

Without redirection to file:

Code: Select all

===========================
Total time (ms) : 40289
Nodes searched  : 48381913
Nodes/second    : 1200871


===========================
Total time (ms) : 40293
Nodes searched  : 48381913
Nodes/second    : 1200752

With redirection to file:

Code: Select all

===========================
Total time (ms) : 40454
Nodes searched  : 48381913
Nodes/second    : 1195973


===========================
Total time (ms) : 40434
Nodes searched  : 48381913
Nodes/second    : 1196565

I used ./stockfish-2.2.2 bench 256 1 16 default depth

Something related to timing seems wrong with your Windows binary. I guess you've been saying that for a while.

He HAS to have some sort of graphics/driver issue. Maybe the world's slowest graphics card or something, or a graphics library with some sort of internal debugging turned on...

Never seen anything like his numbers linux or windows.

[mcostalba]

So is displaying is slower than filewriting

Yes can be, try this to remove the console delay (under Linux/mingw)

Code: Select all

./stockfish bench > /dev/null