A question about hash

[Adam Hair]

I've been testing Spark 0.3 and Doch against various engines under
these conditions:

4'3" time control
Intel E8400 processor @ 2.95 GHz
128 MB hash
Ponder Off
HS Main Book 7 moves
Windows XP x64
Arena 2.01

I cull out games that end in a draw in under 25 moves and games that,
when coming out of book, one side starts with a 1 centipawn advantage
or more.

I became a little concerned when I noticed that in some cases the results
for some engines didn't seem right. Some engines scored about the same
whether using 1 CPU or 2 CPUs. So I am playing Doch vs some of these
engines with 256 MB of hash. Here are the results so far:

Code: Select all

Doch               128 MB & 1 CPU             128 MB & 2 CPU               256 MB & 2 CPU

Bright 0.4a3          27-12-15                   16-16-21                    16-8-19
Crafty 23.0            ----                      32-5-16                     24-5-14
Rybka 2.2n2           9-21-24                    9-25-18                     11-12-20
Stockfish 1.5.1       8-25-20                    6-20-26                     4-24-16
Thinker 5.4d Inert    13-21-19                   9-22-21                     8-16-19
Glaurung 2.2          17-20-17                   17-20-15                     -----
Toga 141 SE           25-13-15                   14-14-24                     -----
-------------------------------------------------------------------------------------------------------

Spark 0.3

Bright 0.4a3          19-8-27                    10-17-27                     -----
Crafty 23.0            ----                      27-7-20                      -----
Rybka 2.2n2           7-28-19                    6-30-18                      -----
Stockfish 1.5.1       9-21-24                    9-22-23                      -----
Thinker 5.4d Inert    17-16-21                   10-21-23                     -----
Glaurung 2.2          18-15-21                   19-17-18                     -----
Toga 141 SE           17-16-21                   14-14-26                     -----
--------------------------------------------------------------------------------------------------------

I know that these are not enough games to really tell what is going on,
and that using a book instead of a common set of starting positions doesn't
help either.

Am I allowing enough hash for these engines with this processor and
time control? It seems that maybe I am handicapping Stockfish, Crafty,
and Glaurung, yet Rybka and Bright are not showing this ( the Rybka
results surprise me ).

If I set up Arena so that all UCI engines use 256 MB of hash, will that
adversely affect engines that do not need that much hash with this
processor and time control?

[bob]

I've been testing Spark 0.3 and Doch against various engines under
these conditions:

4'3" time control
Intel E8400 processor @ 2.95 GHz
128 MB hash
Ponder Off
HS Main Book 7 moves
Windows XP x64
Arena 2.01

I cull out games that end in a draw in under 25 moves and games that,
when coming out of book, one side starts with a 1 centipawn advantage
or more.

I became a little concerned when I noticed that in some cases the results
for some engines didn't seem right. Some engines scored about the same
whether using 1 CPU or 2 CPUs. So I am playing Doch vs some of these
engines with 256 MB of hash. Here are the results so far:

Code: Select all

Doch               128 MB & 1 CPU             128 MB & 2 CPU               256 MB & 2 CPU

Bright 0.4a3          27-12-15                   16-16-21                    16-8-19
Crafty 23.0            ----                      32-5-16                     24-5-14
Rybka 2.2n2           9-21-24                    9-25-18                     11-12-20
Stockfish 1.5.1       8-25-20                    6-20-26                     4-24-16
Thinker 5.4d Inert    13-21-19                   9-22-21                     8-16-19
Glaurung 2.2          17-20-17                   17-20-15                     -----
Toga 141 SE           25-13-15                   14-14-24                     -----
-------------------------------------------------------------------------------------------------------

Spark 0.3

Bright 0.4a3          19-8-27                    10-17-27                     -----
Crafty 23.0            ----                      27-7-20                      -----
Rybka 2.2n2           7-28-19                    6-30-18                      -----
Stockfish 1.5.1       9-21-24                    9-22-23                      -----
Thinker 5.4d Inert    17-16-21                   10-21-23                     -----
Glaurung 2.2          18-15-21                   19-17-18                     -----
Toga 141 SE           17-16-21                   14-14-26                     -----
--------------------------------------------------------------------------------------------------------

I know that these are not enough games to really tell what is going on,
and that using a book instead of a common set of starting positions doesn't
help either.

Am I allowing enough hash for these engines with this processor and
time control? It seems that maybe I am handicapping Stockfish, Crafty,
and Glaurung, yet Rybka and Bright are not showing this ( the Rybka
results surprise me ).

If I set up Arena so that all UCI engines use 256 MB of hash, will that
adversely affect engines that do not need that much hash with this
processor and time control?

I am not sure I follow the above numbers. If Crafty does worse with 2 cpus, something is wrong, although with so few games, the error bar is very large and could explain this. I've run this test several times on the cluster, and 2 cpu crafty gains 50-60 elo over the one cpu version, all else being equal. I have not run long matches using 4 or 8 however...

[Adam Hair]

I've been testing Spark 0.3 and Doch against various engines under
these conditions:

4'3" time control
Intel E8400 processor @ 2.95 GHz
128 MB hash
Ponder Off
HS Main Book 7 moves
Windows XP x64
Arena 2.01

I cull out games that end in a draw in under 25 moves and games that,
when coming out of book, one side starts with a 1 centipawn advantage
or more.

I became a little concerned when I noticed that in some cases the results
for some engines didn't seem right. Some engines scored about the same
whether using 1 CPU or 2 CPUs. So I am playing Doch vs some of these
engines with 256 MB of hash. Here are the results so far:

Code: Select all

Doch               128 MB & 1 CPU             128 MB & 2 CPU               256 MB & 2 CPU

Bright 0.4a3          27-12-15                   16-16-21                    16-8-19
Crafty 23.0            ----                      32-5-16                     24-5-14
Rybka 2.2n2           9-21-24                    9-25-18                     11-12-20
Stockfish 1.5.1       8-25-20                    6-20-26                     4-24-16
Thinker 5.4d Inert    13-21-19                   9-22-21                     8-16-19
Glaurung 2.2          17-20-17                   17-20-15                     -----
Toga 141 SE           25-13-15                   14-14-24                     -----
-------------------------------------------------------------------------------------------------------

Spark 0.3

Bright 0.4a3          19-8-27                    10-17-27                     -----
Crafty 23.0            ----                      27-7-20                      -----
Rybka 2.2n2           7-28-19                    6-30-18                      -----
Stockfish 1.5.1       9-21-24                    9-22-23                      -----
Thinker 5.4d Inert    17-16-21                   10-21-23                     -----
Glaurung 2.2          18-15-21                   19-17-18                     -----
Toga 141 SE           17-16-21                   14-14-26                     -----
--------------------------------------------------------------------------------------------------------

I know that these are not enough games to really tell what is going on,
and that using a book instead of a common set of starting positions doesn't
help either.

Am I allowing enough hash for these engines with this processor and
time control? It seems that maybe I am handicapping Stockfish, Crafty,
and Glaurung, yet Rybka and Bright are not showing this ( the Rybka
results surprise me ).

If I set up Arena so that all UCI engines use 256 MB of hash, will that
adversely affect engines that do not need that much hash with this
processor and time control?

I am not sure I follow the above numbers. If Crafty does worse with 2 cpus, something is wrong, although with so few games, the error bar is very large and could explain this. I've run this test several times on the cluster, and 2 cpu crafty gains 50-60 elo over the one cpu version, all else being equal. I have not run long matches using 4 or 8 however...

I agree that, with the amount of games played, it is hard to ascertain what
might be going on. My question pertains to the amount of hash that I
should allocate for 2 cpu engines with the processor and time control I
am using. For example, am I hamstringing Crafty 23.0 (2 cpu) by only
allowing 128 MB of hash?

As for the above numbers, I apologize if it isn't clear what I was trying to
convey. The first column lists the engines that Doch and Spark played.
The second column denotes the results of Doch,Spark vs the competitors
with each engine allocated 128 MB of hash ( or as close to that number as
possible) and the competitors using 1 cpu. The second column denotes
128 MB of hash and 2 cpus for the competitors. Column 3 is 256 MB and
2 cpus.

I am only running Crafty with 2 cpus, so I am not comparing 1 cpu to 2
cpus as far as Crafty is concerned. But I am wondering if, for Stockfish,
Glaurung, and Rybka, that the similarity of the 1 cpu and 2 cpu results
is do to the hash being limited to 128 MB. The results when I upped the
hash to 256 MB is inconclusive ( not that I really expected clear results
with only 50 games played per engine ).

I could run the number of games necessary to decide for myself if 128 MB
is enough for all engines, but the time needed to do that will leave no
time to do what I've set out to do. So, I am hoping to get information
from people who already know the answer or at least give me a well-
educated guess. Definitely from someone such as yourself.

[bob]

I've been testing Spark 0.3 and Doch against various engines under
these conditions:

4'3" time control
Intel E8400 processor @ 2.95 GHz
128 MB hash
Ponder Off
HS Main Book 7 moves
Windows XP x64
Arena 2.01

I cull out games that end in a draw in under 25 moves and games that,
when coming out of book, one side starts with a 1 centipawn advantage
or more.

I became a little concerned when I noticed that in some cases the results
for some engines didn't seem right. Some engines scored about the same
whether using 1 CPU or 2 CPUs. So I am playing Doch vs some of these
engines with 256 MB of hash. Here are the results so far:

Code: Select all

Doch               128 MB & 1 CPU             128 MB & 2 CPU               256 MB & 2 CPU

Bright 0.4a3          27-12-15                   16-16-21                    16-8-19
Crafty 23.0            ----                      32-5-16                     24-5-14
Rybka 2.2n2           9-21-24                    9-25-18                     11-12-20
Stockfish 1.5.1       8-25-20                    6-20-26                     4-24-16
Thinker 5.4d Inert    13-21-19                   9-22-21                     8-16-19
Glaurung 2.2          17-20-17                   17-20-15                     -----
Toga 141 SE           25-13-15                   14-14-24                     -----
-------------------------------------------------------------------------------------------------------

Spark 0.3

Bright 0.4a3          19-8-27                    10-17-27                     -----
Crafty 23.0            ----                      27-7-20                      -----
Rybka 2.2n2           7-28-19                    6-30-18                      -----
Stockfish 1.5.1       9-21-24                    9-22-23                      -----
Thinker 5.4d Inert    17-16-21                   10-21-23                     -----
Glaurung 2.2          18-15-21                   19-17-18                     -----
Toga 141 SE           17-16-21                   14-14-26                     -----
--------------------------------------------------------------------------------------------------------

I know that these are not enough games to really tell what is going on,
and that using a book instead of a common set of starting positions doesn't
help either.

Am I allowing enough hash for these engines with this processor and
time control? It seems that maybe I am handicapping Stockfish, Crafty,
and Glaurung, yet Rybka and Bright are not showing this ( the Rybka
results surprise me ).

If I set up Arena so that all UCI engines use 256 MB of hash, will that
adversely affect engines that do not need that much hash with this
processor and time control?

I am not sure I follow the above numbers. If Crafty does worse with 2 cpus, something is wrong, although with so few games, the error bar is very large and could explain this. I've run this test several times on the cluster, and 2 cpu crafty gains 50-60 elo over the one cpu version, all else being equal. I have not run long matches using 4 or 8 however...

I agree that, with the amount of games played, it is hard to ascertain what
might be going on. My question pertains to the amount of hash that I
should allocate for 2 cpu engines with the processor and time control I
am using. For example, am I hamstringing Crafty 23.0 (2 cpu) by only
allowing 128 MB of hash?

As for the above numbers, I apologize if it isn't clear what I was trying to
convey. The first column lists the engines that Doch and Spark played.
The second column denotes the results of Doch,Spark vs the competitors
with each engine allocated 128 MB of hash ( or as close to that number as
possible) and the competitors using 1 cpu. The second column denotes
128 MB of hash and 2 cpus for the competitors. Column 3 is 256 MB and
2 cpus.

I am only running Crafty with 2 cpus, so I am not comparing 1 cpu to 2
cpus as far as Crafty is concerned. But I am wondering if, for Stockfish,
Glaurung, and Rybka, that the similarity of the 1 cpu and 2 cpu results
is do to the hash being limited to 128 MB. The results when I upped the
hash to 256 MB is inconclusive ( not that I really expected clear results
with only 50 games played per engine ).

I could run the number of games necessary to decide for myself if 128 MB
is enough for all engines, but the time needed to do that will leave no
time to do what I've set out to do. So, I am hoping to get information
from people who already know the answer or at least give me a well-
educated guess. Definitely from someone such as yourself.

Here's the simplest way to determine reasonable hash: (and this applies only to Crafty, other programs are likely different if they hash q-search nodes).

run crafty's "bench" command using 2 cpus (mt=2) or whatever number your machine will support. Take the NPS number and divide by 4. Take this number and multiply by the average number of seconds per move in the games you are playing. That is a pretty good estimate of the number of hash table entries you need. Multiply by 16 (16 bytes per hash entry) to get the suggested hash table size.

For example, take the 8-core box I run on all the time. Crafty runs about 20M nodes per second. 1/4 of that is 5M. If I am going to play 1 second per move, on average, then I would take 5M * 1 * 16 and set hash=128M. Hash has to be a power of 2 and I would round up to the next size if possible. So hash=128M rather than 80M which would be rounded down to 64M by Crafty.

That will get you in the right ball park. 2-4x larger is not a problem, but the larger you make it, the more you stress the TLB on your processor and you can cause an increase in memory latency when you blow out the TLB too badly, and more physical pages (4KB pages on X86) is the way to blow this. If you want to tune it more accurately, you could pick a dozen positions and run 'em as an epd test suite, and use the hash=nM command to adjust hash and choose the size that produces the best overall search depth for a fixed target time.

[bob]

I've been testing Spark 0.3 and Doch against various engines under
these conditions:

4'3" time control
Intel E8400 processor @ 2.95 GHz
128 MB hash
Ponder Off
HS Main Book 7 moves
Windows XP x64
Arena 2.01

I cull out games that end in a draw in under 25 moves and games that,
when coming out of book, one side starts with a 1 centipawn advantage
or more.

I became a little concerned when I noticed that in some cases the results
for some engines didn't seem right. Some engines scored about the same
whether using 1 CPU or 2 CPUs. So I am playing Doch vs some of these
engines with 256 MB of hash. Here are the results so far:

Code: Select all

Doch               128 MB & 1 CPU             128 MB & 2 CPU               256 MB & 2 CPU

Bright 0.4a3          27-12-15                   16-16-21                    16-8-19
Crafty 23.0            ----                      32-5-16                     24-5-14
Rybka 2.2n2           9-21-24                    9-25-18                     11-12-20
Stockfish 1.5.1       8-25-20                    6-20-26                     4-24-16
Thinker 5.4d Inert    13-21-19                   9-22-21                     8-16-19
Glaurung 2.2          17-20-17                   17-20-15                     -----
Toga 141 SE           25-13-15                   14-14-24                     -----
-------------------------------------------------------------------------------------------------------

Spark 0.3

Bright 0.4a3          19-8-27                    10-17-27                     -----
Crafty 23.0            ----                      27-7-20                      -----
Rybka 2.2n2           7-28-19                    6-30-18                      -----
Stockfish 1.5.1       9-21-24                    9-22-23                      -----
Thinker 5.4d Inert    17-16-21                   10-21-23                     -----
Glaurung 2.2          18-15-21                   19-17-18                     -----
Toga 141 SE           17-16-21                   14-14-26                     -----
--------------------------------------------------------------------------------------------------------

I know that these are not enough games to really tell what is going on,
and that using a book instead of a common set of starting positions doesn't
help either.

Am I allowing enough hash for these engines with this processor and
time control? It seems that maybe I am handicapping Stockfish, Crafty,
and Glaurung, yet Rybka and Bright are not showing this ( the Rybka
results surprise me ).

If I set up Arena so that all UCI engines use 256 MB of hash, will that
adversely affect engines that do not need that much hash with this
processor and time control?

I am not sure I follow the above numbers. If Crafty does worse with 2 cpus, something is wrong, although with so few games, the error bar is very large and could explain this. I've run this test several times on the cluster, and 2 cpu crafty gains 50-60 elo over the one cpu version, all else being equal. I have not run long matches using 4 or 8 however...

I agree that, with the amount of games played, it is hard to ascertain what
might be going on. My question pertains to the amount of hash that I
should allocate for 2 cpu engines with the processor and time control I
am using. For example, am I hamstringing Crafty 23.0 (2 cpu) by only
allowing 128 MB of hash?

As for the above numbers, I apologize if it isn't clear what I was trying to
convey. The first column lists the engines that Doch and Spark played.
The second column denotes the results of Doch,Spark vs the competitors
with each engine allocated 128 MB of hash ( or as close to that number as
possible) and the competitors using 1 cpu. The second column denotes
128 MB of hash and 2 cpus for the competitors. Column 3 is 256 MB and
2 cpus.

I am only running Crafty with 2 cpus, so I am not comparing 1 cpu to 2
cpus as far as Crafty is concerned. But I am wondering if, for Stockfish,
Glaurung, and Rybka, that the similarity of the 1 cpu and 2 cpu results
is do to the hash being limited to 128 MB. The results when I upped the
hash to 256 MB is inconclusive ( not that I really expected clear results
with only 50 games played per engine ).

I could run the number of games necessary to decide for myself if 128 MB
is enough for all engines, but the time needed to do that will leave no
time to do what I've set out to do. So, I am hoping to get information
from people who already know the answer or at least give me a well-
educated guess. Definitely from someone such as yourself.

Here's the simplest way to determine reasonable hash: (and this applies only to Crafty, other programs are likely different if they hash q-search nodes).

run crafty's "bench" command using 2 cpus (mt=2) or whatever number your machine will support. Take the NPS number and divide by 4. Take this number and multiply by the average number of seconds per move in the games you are playing. That is a pretty good estimate of the number of hash table entries you need. Multiply by 16 (16 bytes per hash entry) to get the suggested hash table size.

For example, take the 8-core box I run on all the time. Crafty runs about 20M nodes per second. 1/4 of that is 5M. If I am going to play 1 second per move, on average, then I would take 5M * 1 * 16 and set hash=128M. Hash has to be a power of 2 and I would round up to the next size if possible. So hash=128M rather than 80M which would be rounded down to 64M by Crafty.

That will get you in the right ball park. 2-4x larger is not a problem, but the larger you make it, the more you stress the TLB on your processor and you can cause an increase in memory latency when you blow out the TLB too badly, and more physical pages (4KB pages on X86) is the way to blow this. If you want to tune it more accurately, you could pick a dozen positions and run 'em as an epd test suite, and use the hash=nM command to adjust hash and choose the size that produces the best overall search depth for a fixed target time.

One key. If you tell crafty to use 2 threads, you +must+ have two idle processors. They can not be used by another program, not even one of them, or it will totally wreck Crafty's parallel search performance...

[Adam Hair]

I've been testing Spark 0.3 and Doch against various engines under
these conditions:

4'3" time control
Intel E8400 processor @ 2.95 GHz
128 MB hash
Ponder Off
HS Main Book 7 moves
Windows XP x64
Arena 2.01

I cull out games that end in a draw in under 25 moves and games that,
when coming out of book, one side starts with a 1 centipawn advantage
or more.

I became a little concerned when I noticed that in some cases the results
for some engines didn't seem right. Some engines scored about the same
whether using 1 CPU or 2 CPUs. So I am playing Doch vs some of these
engines with 256 MB of hash. Here are the results so far:

Code: Select all

Doch               128 MB & 1 CPU             128 MB & 2 CPU               256 MB & 2 CPU

Bright 0.4a3          27-12-15                   16-16-21                    16-8-19
Crafty 23.0            ----                      32-5-16                     24-5-14
Rybka 2.2n2           9-21-24                    9-25-18                     11-12-20
Stockfish 1.5.1       8-25-20                    6-20-26                     4-24-16
Thinker 5.4d Inert    13-21-19                   9-22-21                     8-16-19
Glaurung 2.2          17-20-17                   17-20-15                     -----
Toga 141 SE           25-13-15                   14-14-24                     -----
-------------------------------------------------------------------------------------------------------

Spark 0.3

Bright 0.4a3          19-8-27                    10-17-27                     -----
Crafty 23.0            ----                      27-7-20                      -----
Rybka 2.2n2           7-28-19                    6-30-18                      -----
Stockfish 1.5.1       9-21-24                    9-22-23                      -----
Thinker 5.4d Inert    17-16-21                   10-21-23                     -----
Glaurung 2.2          18-15-21                   19-17-18                     -----
Toga 141 SE           17-16-21                   14-14-26                     -----
--------------------------------------------------------------------------------------------------------

I know that these are not enough games to really tell what is going on,
and that using a book instead of a common set of starting positions doesn't
help either.

Am I allowing enough hash for these engines with this processor and
time control? It seems that maybe I am handicapping Stockfish, Crafty,
and Glaurung, yet Rybka and Bright are not showing this ( the Rybka
results surprise me ).

If I set up Arena so that all UCI engines use 256 MB of hash, will that
adversely affect engines that do not need that much hash with this
processor and time control?

I am not sure I follow the above numbers. If Crafty does worse with 2 cpus, something is wrong, although with so few games, the error bar is very large and could explain this. I've run this test several times on the cluster, and 2 cpu crafty gains 50-60 elo over the one cpu version, all else being equal. I have not run long matches using 4 or 8 however...

I agree that, with the amount of games played, it is hard to ascertain what
might be going on. My question pertains to the amount of hash that I
should allocate for 2 cpu engines with the processor and time control I
am using. For example, am I hamstringing Crafty 23.0 (2 cpu) by only
allowing 128 MB of hash?

As for the above numbers, I apologize if it isn't clear what I was trying to
convey. The first column lists the engines that Doch and Spark played.
The second column denotes the results of Doch,Spark vs the competitors
with each engine allocated 128 MB of hash ( or as close to that number as
possible) and the competitors using 1 cpu. The second column denotes
128 MB of hash and 2 cpus for the competitors. Column 3 is 256 MB and
2 cpus.

I am only running Crafty with 2 cpus, so I am not comparing 1 cpu to 2
cpus as far as Crafty is concerned. But I am wondering if, for Stockfish,
Glaurung, and Rybka, that the similarity of the 1 cpu and 2 cpu results
is do to the hash being limited to 128 MB. The results when I upped the
hash to 256 MB is inconclusive ( not that I really expected clear results
with only 50 games played per engine ).

I could run the number of games necessary to decide for myself if 128 MB
is enough for all engines, but the time needed to do that will leave no
time to do what I've set out to do. So, I am hoping to get information
from people who already know the answer or at least give me a well-
educated guess. Definitely from someone such as yourself.

Here's the simplest way to determine reasonable hash: (and this applies only to Crafty, other programs are likely different if they hash q-search nodes).

run crafty's "bench" command using 2 cpus (mt=2) or whatever number your machine will support. Take the NPS number and divide by 4. Take this number and multiply by the average number of seconds per move in the games you are playing. That is a pretty good estimate of the number of hash table entries you need. Multiply by 16 (16 bytes per hash entry) to get the suggested hash table size.

For example, take the 8-core box I run on all the time. Crafty runs about 20M nodes per second. 1/4 of that is 5M. If I am going to play 1 second per move, on average, then I would take 5M * 1 * 16 and set hash=128M. Hash has to be a power of 2 and I would round up to the next size if possible. So hash=128M rather than 80M which would be rounded down to 64M by Crafty.

That will get you in the right ball park. 2-4x larger is not a problem, but the larger you make it, the more you stress the TLB on your processor and you can cause an increase in memory latency when you blow out the TLB too badly, and more physical pages (4KB pages on X86) is the way to blow this. If you want to tune it more accurately, you could pick a dozen positions and run 'em as an epd test suite, and use the hash=nM command to adjust hash and choose the size that produces the best overall search depth for a fixed target time.

Thanks Bob. I will do this.