Page 1 of 3

Arasan test suite with minimal depth

Posted: Thu May 10, 2018 12:16 pm
by xr_a_y
Can someone post the Arasan test suite with minimal depth needed to solve each position ?

Re: Arasan test suite with minimal depth

Posted: Thu May 10, 2018 4:13 pm
by jdart
The depth required is going to vary depending on the engine, and if multithreaded, it can also vary from one run to the next.

--Jon

Re: Arasan test suite with minimal depth

Posted: Thu May 10, 2018 4:56 pm
by xr_a_y
Yes, but there is some kind of minimal depth under which the good move cannot probably be found (except by "luck").

Re: Arasan test suite with minimal depth

Posted: Thu May 10, 2018 6:20 pm
by Dann Corbit
If everyone used pure alpha-beta with no other pruning or extensions and the same evaluation, then there would be a clear answer.

Sting finds the answer to this position in 4 seconds:
5rk1/pp3p2/7b/2pR4/8/2P4P/P1PNr1P1/2K4R b - -
Stockfish did not find it in 1/2 hour on my machine with the same tablebase, threadcount, hash, etc.
But Stockfish is clearly stronger than Sting.

That is why the question is so difficult.

Re: Arasan test suite with minimal depth

Posted: Thu May 10, 2018 6:39 pm
by xr_a_y
I was looking for some kind of minimal depth because at 60sec per position, my engine is only scoring 10/200, and I know I really miss some better pruning to go deeper. So if most positions need deep analysis to find the good move (and by deep, I mean more than 12 plies) , I will relaunch the test with 30 min per positions.

Re: Arasan test suite with minimal depth

Posted: Thu May 10, 2018 6:44 pm
by Dann Corbit
Did you debug your evaluation by rotation of the board and reversal of the colors to ensure that it is 100% correct?
I guess that you have a defect.

Re: Arasan test suite with minimal depth

Posted: Thu May 10, 2018 6:53 pm
by Dann Corbit
Grouped by 4, each position (of the 4) should have identical evaluation. If not, then there is an evaluation bug.

Re: Arasan test suite with minimal depth

Posted: Thu May 10, 2018 11:05 pm
by jdart
xr_a_y wrote:
Thu May 10, 2018 6:39 pm
My engine is only scoring 10/200, and I know I really miss some better pruning to go deeper. ,So if most positions need deep analysis to find the good move (and by deep, I mean more than 12 plies)
If you are not getting a lot more than 12 plies at 60 seconds/position, then something is badly wrong. Maybe a very bad branching factor or more likely bugs.

These are medium-hard positions (harder than WAC for example) but mostly not very challenging for top-level engines. Arasan doesn't actually do so well on these itself, but the latest version scored 112/200 at 60 seconds/position, on a quad core system.

--Jon

Re: Arasan test suite with minimal depth

Posted: Fri May 11, 2018 4:57 am
by xr_a_y
Dann Corbit wrote:
Thu May 10, 2018 6:53 pm
Grouped by 4, each position (of the 4) should have identical evaluation. If not, then there is an evaluation bug.
Thanks I'll give it a shot today.

Re: Arasan test suite with minimal depth

Posted: Fri May 11, 2018 5:02 am
by xr_a_y
jdart wrote:
Thu May 10, 2018 11:05 pm
xr_a_y wrote:
Thu May 10, 2018 6:39 pm
My engine is only scoring 10/200, and I know I really miss some better pruning to go deeper. ,So if most positions need deep analysis to find the good move (and by deep, I mean more than 12 plies)
If you are not getting a lot more than 12 plies at 60 seconds/position, then something is badly wrong. Maybe a very bad branching factor or more likely bugs.

These are medium-hard positions (harder than WAC for example) but mostly not very challenging for top-level engines. Arasan doesn't actually do so well on these itself, but the latest version scored 112/200 at 60 seconds/position, on a quad core system.

--Jon
I am always looking for bugs of course ... But note that microMax is not going more than 12 plies in 60sec and has probably no horrible bug.