Arasan test suite with minimal depth

[xr_a_y]

Can someone post the Arasan test suite with minimal depth needed to solve each position ?

[jdart]

The depth required is going to vary depending on the engine, and if multithreaded, it can also vary from one run to the next.

--Jon

[xr_a_y]

Yes, but there is some kind of minimal depth under which the good move cannot probably be found (except by "luck").

[Dann Corbit]

If everyone used pure alpha-beta with no other pruning or extensions and the same evaluation, then there would be a clear answer.

Sting finds the answer to this position in 4 seconds:
5rk1/pp3p2/7b/2pR4/8/2P4P/P1PNr1P1/2K4R b - -
Stockfish did not find it in 1/2 hour on my machine with the same tablebase, threadcount, hash, etc.
But Stockfish is clearly stronger than Sting.

That is why the question is so difficult.

[xr_a_y]

I was looking for some kind of minimal depth because at 60sec per position, my engine is only scoring 10/200, and I know I really miss some better pruning to go deeper. So if most positions need deep analysis to find the good move (and by deep, I mean more than 12 plies) , I will relaunch the test with 30 min per positions.

[Dann Corbit]

Did you debug your evaluation by rotation of the board and reversal of the colors to ensure that it is 100% correct?
I guess that you have a defect.

[Dann Corbit]

Grouped by 4, each position (of the 4) should have identical evaluation. If not, then there is an evaluation bug.

[jdart]

My engine is only scoring 10/200, and I know I really miss some better pruning to go deeper. ,So if most positions need deep analysis to find the good move (and by deep, I mean more than 12 plies)

If you are not getting a lot more than 12 plies at 60 seconds/position, then something is badly wrong. Maybe a very bad branching factor or more likely bugs.

These are medium-hard positions (harder than WAC for example) but mostly not very challenging for top-level engines. Arasan doesn't actually do so well on these itself, but the latest version scored 112/200 at 60 seconds/position, on a quad core system.

--Jon

[xr_a_y]

Grouped by 4, each position (of the 4) should have identical evaluation. If not, then there is an evaluation bug.

Thanks I'll give it a shot today.

[xr_a_y]

My engine is only scoring 10/200, and I know I really miss some better pruning to go deeper. ,So if most positions need deep analysis to find the good move (and by deep, I mean more than 12 plies)

If you are not getting a lot more than 12 plies at 60 seconds/position, then something is badly wrong. Maybe a very bad branching factor or more likely bugs.

These are medium-hard positions (harder than WAC for example) but mostly not very challenging for top-level engines. Arasan doesn't actually do so well on these itself, but the latest version scored 112/200 at 60 seconds/position, on a quad core system.

--Jon

I am always looking for bugs of course ... But note that microMax is not going more than 12 plies in 60sec and has probably no horrible bug.