Arasan test suite with minimal depth
Moderators: hgm, Rebel, chrisw
-
- Posts: 1871
- Joined: Sat Nov 25, 2017 2:28 pm
- Location: France
Arasan test suite with minimal depth
Can someone post the Arasan test suite with minimal depth needed to solve each position ?
-
- Posts: 4367
- Joined: Fri Mar 10, 2006 5:23 am
- Location: http://www.arasanchess.org
Re: Arasan test suite with minimal depth
The depth required is going to vary depending on the engine, and if multithreaded, it can also vary from one run to the next.
--Jon
--Jon
-
- Posts: 1871
- Joined: Sat Nov 25, 2017 2:28 pm
- Location: France
Re: Arasan test suite with minimal depth
Yes, but there is some kind of minimal depth under which the good move cannot probably be found (except by "luck").
-
- Posts: 12542
- Joined: Wed Mar 08, 2006 8:57 pm
- Location: Redmond, WA USA
Re: Arasan test suite with minimal depth
If everyone used pure alpha-beta with no other pruning or extensions and the same evaluation, then there would be a clear answer.
Sting finds the answer to this position in 4 seconds:
5rk1/pp3p2/7b/2pR4/8/2P4P/P1PNr1P1/2K4R b - -
Stockfish did not find it in 1/2 hour on my machine with the same tablebase, threadcount, hash, etc.
But Stockfish is clearly stronger than Sting.
That is why the question is so difficult.
Sting finds the answer to this position in 4 seconds:
5rk1/pp3p2/7b/2pR4/8/2P4P/P1PNr1P1/2K4R b - -
Stockfish did not find it in 1/2 hour on my machine with the same tablebase, threadcount, hash, etc.
But Stockfish is clearly stronger than Sting.
That is why the question is so difficult.
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
-
- Posts: 1871
- Joined: Sat Nov 25, 2017 2:28 pm
- Location: France
Re: Arasan test suite with minimal depth
I was looking for some kind of minimal depth because at 60sec per position, my engine is only scoring 10/200, and I know I really miss some better pruning to go deeper. So if most positions need deep analysis to find the good move (and by deep, I mean more than 12 plies) , I will relaunch the test with 30 min per positions.
-
- Posts: 12542
- Joined: Wed Mar 08, 2006 8:57 pm
- Location: Redmond, WA USA
Re: Arasan test suite with minimal depth
Did you debug your evaluation by rotation of the board and reversal of the colors to ensure that it is 100% correct?
I guess that you have a defect.
I guess that you have a defect.
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
-
- Posts: 12542
- Joined: Wed Mar 08, 2006 8:57 pm
- Location: Redmond, WA USA
Re: Arasan test suite with minimal depth
Grouped by 4, each position (of the 4) should have identical evaluation. If not, then there is an evaluation bug.
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
-
- Posts: 4367
- Joined: Fri Mar 10, 2006 5:23 am
- Location: http://www.arasanchess.org
Re: Arasan test suite with minimal depth
If you are not getting a lot more than 12 plies at 60 seconds/position, then something is badly wrong. Maybe a very bad branching factor or more likely bugs.
These are medium-hard positions (harder than WAC for example) but mostly not very challenging for top-level engines. Arasan doesn't actually do so well on these itself, but the latest version scored 112/200 at 60 seconds/position, on a quad core system.
--Jon
-
- Posts: 1871
- Joined: Sat Nov 25, 2017 2:28 pm
- Location: France
Re: Arasan test suite with minimal depth
Thanks I'll give it a shot today.Dann Corbit wrote: ↑Thu May 10, 2018 8:53 pm Grouped by 4, each position (of the 4) should have identical evaluation. If not, then there is an evaluation bug.
-
- Posts: 1871
- Joined: Sat Nov 25, 2017 2:28 pm
- Location: France
Re: Arasan test suite with minimal depth
I am always looking for bugs of course ... But note that microMax is not going more than 12 plies in 60sec and has probably no horrible bug.jdart wrote: ↑Fri May 11, 2018 1:05 amIf you are not getting a lot more than 12 plies at 60 seconds/position, then something is badly wrong. Maybe a very bad branching factor or more likely bugs.
These are medium-hard positions (harder than WAC for example) but mostly not very challenging for top-level engines. Arasan doesn't actually do so well on these itself, but the latest version scored 112/200 at 60 seconds/position, on a quad core system.
--Jon