'STS' Test Suite - Available for Testing

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
swami
Posts: 6535
Joined: Thu Mar 09, 2006 3:21 am

'STS' Test Suite - Available for Testing

Post by swami » Sat Nov 15, 2008 4:53 pm

Strategic Test Suite (STS) v1.0 is available for testing:

Beta version of the Test suite file will consist of 100 selected Undermining/Pawn Weakening positions - along with solutions.


It's still too early for me to release it, I'd probably release it once I get the test suite almost error- free. That's why I'm looking for prospective authors/testers willing to test their engines using it and report the results/ bugs back to me.

Please either send me a pm here or send me an e-mail (nswami15 at yahoo dot com) to obtain this test suite.

Thanks in advance.

Ps; This test suite is ideal for engines <2900.

swami
Posts: 6535
Joined: Thu Mar 09, 2006 3:21 am

Re: 'STS' Test Suite - Available for Testing

Post by swami » Sat Nov 15, 2008 5:25 pm

I'm working on building up the test suites that cover these complex strategic principles:

Undermining/Pawn Weakening (Completed!)
Blockade
Outpost
Prophylxaxis
Counter Play
Simplification
Positional Sacrifice
Weaknesses
Initiative

This is getting painstakingly slow... So If there's anyone who is willing to work as a team, then I would appreciate it and would give them credit for the work.

swami
Posts: 6535
Joined: Thu Mar 09, 2006 3:21 am

Re: 'STS' Test Suite - Available for Testing

Post by swami » Sun Nov 16, 2008 5:57 am

swami wrote:Ps; This test suite is ideal for engines <2900.
This ain't suited to Rybka. Since this was _mostly_ basically taken from Rybka 3.0 4 CPU games! I think this is more ideal for testing engines other than Rybka.

I selected positions where pawn weakening/undermining happens in Rybka games. So essentially all of the 100 positions are basically _pawn_ moves.

Cut off was > +0.20. If difference in Rybka 2.2n2's analysis line from first best to second best is < 0.20 after about 1 minute of analysis then I'd not add it to the test suite. Only If it's >0.20 (and significant) then I'd add it.

Conditions for the test suite:

Time: anywhere ranging from 30 seconds to 1 minute each position.

swami
Posts: 6535
Joined: Thu Mar 09, 2006 3:21 am

Re: 'STS' Test Suite - Available for Testing

Post by swami » Sun Nov 30, 2008 10:30 am

Re: Test Suite Progress:

Dann Corbitt is now in my team, hip hip hoooray! :D

He has been kind, hardworking, and has refined the test suite to include only sensible test positions. He says he found the test suite to be intensely exciting! :)

We have nearly finished the Undermining chapter. Now it is in the final testing stage, I will release it once that is done.

User avatar
Kempelen
Posts: 620
Joined: Fri Feb 08, 2008 9:44 am
Location: Madrid - Spain
Contact:

Re: 'STS' Test Suite - Available for Testing

Post by Kempelen » Wed Jan 07, 2009 11:23 am

I have just read this topic. I had the same idea from long time ago, but I have not found time to execute it.
In what point is the work? I'm expecting
Fermin Serrano
Author of 'Rodin' engine
http://sites.google.com/site/clonfsp/

swami
Posts: 6535
Joined: Thu Mar 09, 2006 3:21 am

Re: 'STS' Test Suite - Available for Testing

Post by swami » Wed Jan 07, 2009 12:04 pm

Kempelen wrote:I have just read this topic. I had the same idea from long time ago, but I have not found time to execute it.
In what point is the work? I'm expecting
Hi Fermin,

I've decided to release the test suite finally, this was planned to be released on New Year's but got delayed a bit.

Please download here:
http://computerchessblogger.googlepages ... tional.rar

Regards.

Jouni
Posts: 1956
Joined: Wed Mar 08, 2006 7:15 pm

Re: 'STS' Test Suite - Available for Testing

Post by Jouni » Wed Jan 07, 2009 3:10 pm

Finally new testsuite, thanks! But isn't this TOO EASY: Naum and Grapefruit (1 CPU) got 90-91/100 with one minute limit only.

Jouni

swami
Posts: 6535
Joined: Thu Mar 09, 2006 3:21 am

Re: 'STS' Test Suite - Available for Testing

Post by swami » Wed Jan 07, 2009 3:31 pm

Jouni wrote:Finally new testsuite, thanks! But isn't this TOO EASY: Naum and Grapefruit (1 CPU) got 90-91/100 with one minute limit only.

Jouni
It could be easy for really stronger engines but do not forget that: Main objective is that it's designed and is more suitable to engines ranging from 1500-2700. Rybka, Naum and Fruit can solve many.

I'm surprised that they solved only 90/100 with 1 minute limit. They would probably have solved 100/100 in tactics. Anyway Thanks for posting the results. :)

kgburcham
Posts: 2016
Joined: Sun Feb 17, 2008 3:19 pm

compare 5 programs

Post by kgburcham » Wed Jan 07, 2009 4:35 pm

It is interesting to see how Rybka compares to other programs on same hardware. In these types of positions Rybka has a big advantage.

STS Test Suite
Q6600 4x2.4
Hash 1024

time to solve=160 seconds


Rybka 3
Total Time 6:01m Solution Time: 2:11m

Zappa Mexico II
TotTime: 8:20m SolTime: 6:25m

Deep Sjeng WC2008 x64
TotTime: 19:27m SolTime: 16:38m

Deep Shredder 11 x64
TotTime: 20:07m SolTime: 18:08m

Deep Hiarcs Paderborn
88 out of 100. Average time = 4.77s / 10.98

Cubeman
Posts: 644
Joined: Fri Feb 02, 2007 2:11 am
Location: New Zealand

Re: compare 5 programs

Post by Cubeman » Thu Jan 08, 2009 7:32 am

Is there a PGN file of this test.I have a few programs on my PPC and can only use PGN.

Post Reply