Page 1 of 3
Arasan test suite with minimal depth
Posted: Thu May 10, 2018 2:16 pm
by xr_a_y
Can someone post the Arasan test suite with minimal depth needed to solve each position ?
Re: Arasan test suite with minimal depth
Posted: Thu May 10, 2018 6:13 pm
by jdart
The depth required is going to vary depending on the engine, and if multithreaded, it can also vary from one run to the next.
--Jon
Re: Arasan test suite with minimal depth
Posted: Thu May 10, 2018 6:56 pm
by xr_a_y
Yes, but there is some kind of minimal depth under which the good move cannot probably be found (except by "luck").
Re: Arasan test suite with minimal depth
Posted: Thu May 10, 2018 8:20 pm
by Dann Corbit
If everyone used pure alpha-beta with no other pruning or extensions and the same evaluation, then there would be a clear answer.
Sting finds the answer to this position in 4 seconds:
5rk1/pp3p2/7b/2pR4/8/2P4P/P1PNr1P1/2K4R b - -
Stockfish did not find it in 1/2 hour on my machine with the same tablebase, threadcount, hash, etc.
But Stockfish is clearly stronger than Sting.
That is why the question is so difficult.
Re: Arasan test suite with minimal depth
Posted: Thu May 10, 2018 8:39 pm
by xr_a_y
I was looking for some kind of minimal depth because at 60sec per position, my engine is only scoring 10/200, and I know I really miss some better pruning to go deeper. So if most positions need deep analysis to find the good move (and by deep, I mean more than 12 plies) , I will relaunch the test with 30 min per positions.
Re: Arasan test suite with minimal depth
Posted: Thu May 10, 2018 8:44 pm
by Dann Corbit
Did you debug your evaluation by rotation of the board and reversal of the colors to ensure that it is 100% correct?
I guess that you have a defect.
Re: Arasan test suite with minimal depth
Posted: Thu May 10, 2018 8:53 pm
by Dann Corbit
Grouped by 4, each position (of the 4) should have identical evaluation. If not, then there is an evaluation bug.
Re: Arasan test suite with minimal depth
Posted: Fri May 11, 2018 1:05 am
by jdart
xr_a_y wrote: ↑Thu May 10, 2018 8:39 pm
My engine is only scoring 10/200, and I know I really miss some better pruning to go deeper. ,So if most positions need deep analysis to find the good move (and by deep, I mean more than 12 plies)
If you are not getting a lot more than 12 plies at 60 seconds/position, then something is badly wrong. Maybe a very bad branching factor or more likely bugs.
These are medium-hard positions (harder than WAC for example) but mostly not very challenging for top-level engines. Arasan doesn't actually do so well on these itself, but the latest version scored 112/200 at 60 seconds/position, on a quad core system.
--Jon
Re: Arasan test suite with minimal depth
Posted: Fri May 11, 2018 6:57 am
by xr_a_y
Dann Corbit wrote: ↑Thu May 10, 2018 8:53 pm
Grouped by 4, each position (of the 4) should have identical evaluation. If not, then there is an evaluation bug.
Thanks I'll give it a shot today.
Re: Arasan test suite with minimal depth
Posted: Fri May 11, 2018 7:02 am
by xr_a_y
jdart wrote: ↑Fri May 11, 2018 1:05 am
xr_a_y wrote: ↑Thu May 10, 2018 8:39 pm
My engine is only scoring 10/200, and I know I really miss some better pruning to go deeper. ,So if most positions need deep analysis to find the good move (and by deep, I mean more than 12 plies)
If you are not getting a lot more than 12 plies at 60 seconds/position, then something is badly wrong. Maybe a very bad branching factor or more likely bugs.
These are medium-hard positions (harder than WAC for example) but mostly not very challenging for top-level engines. Arasan doesn't actually do so well on these itself, but the latest version scored 112/200 at 60 seconds/position, on a quad core system.
--Jon
I am always looking for bugs of course ... But note that microMax is not going more than 12 plies in 60sec and has probably no horrible bug.