Arasan test suite with minimal depth

Discussion of chess software programming and technical issues.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
xr_a_y
Posts: 236
Joined: Sat Nov 25, 2017 1:28 pm
Location: France

Arasan test suite with minimal depth

Post by xr_a_y » Thu May 10, 2018 12:16 pm

Can someone post the Arasan test suite with minimal depth needed to solve each position ?

jdart
Posts: 3585
Joined: Fri Mar 10, 2006 4:23 am
Location: http://www.arasanchess.org

Re: Arasan test suite with minimal depth

Post by jdart » Thu May 10, 2018 4:13 pm

The depth required is going to vary depending on the engine, and if multithreaded, it can also vary from one run to the next.

--Jon

User avatar
xr_a_y
Posts: 236
Joined: Sat Nov 25, 2017 1:28 pm
Location: France

Re: Arasan test suite with minimal depth

Post by xr_a_y » Thu May 10, 2018 4:56 pm

Yes, but there is some kind of minimal depth under which the good move cannot probably be found (except by "luck").

Dann Corbit
Posts: 8953
Joined: Wed Mar 08, 2006 7:57 pm
Location: Redmond, WA USA
Contact:

Re: Arasan test suite with minimal depth

Post by Dann Corbit » Thu May 10, 2018 6:20 pm

If everyone used pure alpha-beta with no other pruning or extensions and the same evaluation, then there would be a clear answer.

Sting finds the answer to this position in 4 seconds:
5rk1/pp3p2/7b/2pR4/8/2P4P/P1PNr1P1/2K4R b - -
Stockfish did not find it in 1/2 hour on my machine with the same tablebase, threadcount, hash, etc.
But Stockfish is clearly stronger than Sting.

That is why the question is so difficult.
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.

User avatar
xr_a_y
Posts: 236
Joined: Sat Nov 25, 2017 1:28 pm
Location: France

Re: Arasan test suite with minimal depth

Post by xr_a_y » Thu May 10, 2018 6:39 pm

I was looking for some kind of minimal depth because at 60sec per position, my engine is only scoring 10/200, and I know I really miss some better pruning to go deeper. So if most positions need deep analysis to find the good move (and by deep, I mean more than 12 plies) , I will relaunch the test with 30 min per positions.

Dann Corbit
Posts: 8953
Joined: Wed Mar 08, 2006 7:57 pm
Location: Redmond, WA USA
Contact:

Re: Arasan test suite with minimal depth

Post by Dann Corbit » Thu May 10, 2018 6:44 pm

Did you debug your evaluation by rotation of the board and reversal of the colors to ensure that it is 100% correct?
I guess that you have a defect.
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.

Dann Corbit
Posts: 8953
Joined: Wed Mar 08, 2006 7:57 pm
Location: Redmond, WA USA
Contact:

Re: Arasan test suite with minimal depth

Post by Dann Corbit » Thu May 10, 2018 6:53 pm

Grouped by 4, each position (of the 4) should have identical evaluation. If not, then there is an evaluation bug.
Attachments
MC_OP_arasan.7z
Arasan test suite, with rotation and flip
(21.98 KiB) Downloaded 61 times
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.

jdart
Posts: 3585
Joined: Fri Mar 10, 2006 4:23 am
Location: http://www.arasanchess.org

Re: Arasan test suite with minimal depth

Post by jdart » Thu May 10, 2018 11:05 pm

xr_a_y wrote:
Thu May 10, 2018 6:39 pm
My engine is only scoring 10/200, and I know I really miss some better pruning to go deeper. ,So if most positions need deep analysis to find the good move (and by deep, I mean more than 12 plies)
If you are not getting a lot more than 12 plies at 60 seconds/position, then something is badly wrong. Maybe a very bad branching factor or more likely bugs.

These are medium-hard positions (harder than WAC for example) but mostly not very challenging for top-level engines. Arasan doesn't actually do so well on these itself, but the latest version scored 112/200 at 60 seconds/position, on a quad core system.

--Jon

User avatar
xr_a_y
Posts: 236
Joined: Sat Nov 25, 2017 1:28 pm
Location: France

Re: Arasan test suite with minimal depth

Post by xr_a_y » Fri May 11, 2018 4:57 am

Dann Corbit wrote:
Thu May 10, 2018 6:53 pm
Grouped by 4, each position (of the 4) should have identical evaluation. If not, then there is an evaluation bug.
Thanks I'll give it a shot today.

User avatar
xr_a_y
Posts: 236
Joined: Sat Nov 25, 2017 1:28 pm
Location: France

Re: Arasan test suite with minimal depth

Post by xr_a_y » Fri May 11, 2018 5:02 am

jdart wrote:
Thu May 10, 2018 11:05 pm
xr_a_y wrote:
Thu May 10, 2018 6:39 pm
My engine is only scoring 10/200, and I know I really miss some better pruning to go deeper. ,So if most positions need deep analysis to find the good move (and by deep, I mean more than 12 plies)
If you are not getting a lot more than 12 plies at 60 seconds/position, then something is badly wrong. Maybe a very bad branching factor or more likely bugs.

These are medium-hard positions (harder than WAC for example) but mostly not very challenging for top-level engines. Arasan doesn't actually do so well on these itself, but the latest version scored 112/200 at 60 seconds/position, on a quad core system.

--Jon
I am always looking for bugs of course ... But note that microMax is not going more than 12 plies in 60sec and has probably no horrible bug.

Post Reply