I'm using Arena 1.1. If you recall, i had problems with STS suites getting a different score when repeating the test. Perhaps this was what was happening to me. At any rate, i had to stop using STS for this reason.
Regards
Dave
My guess is that engines are really dynamic in their search when the time given is only 10 seconds per move. It would be best if they remain static so that we could know what exactly the scores were.
perhaps you could try the tests with shredder GUI/Chessbase GUI/ChessGUI and different engines and see if it does the same?
Doesn't STSStat only calculate the scores from an Arena .log file? So how would the scores be calculated when using any of these other gui to run the tests?
David Dahlem wrote:Doesn't STSStat only calculate the scores from an Arena .log file? So how would the scores be calculated when using any of these other gui to run the tests?
Yeah, but ChessGUI has its own calculation. There's partial credit scores calculation as well. Don't know about Shredder/Aquarium/ChessbaseGUI's.
You could also try increasing the time to say 30 seconds per position and see if the results are static in Arena.
jwes wrote:
No, I am saying that Arena sometimes adds a second solution to a position, and it can happen that programs can find this second move, and be counted right, though that move is not a solution given in the epd record.
Yes, I saw that. Arena appears to be adding extra moves the most in STS 2 and STS 3 (nearly 10 positions). Not sure where the move came from?
I guess I'll have to use ChessGUI from now onwards.
Have you also tried Arena 1.1? It's an old supposedly stable version.
Tests may take longer in ChessGUI because of its basic philosophy:
after each game or test position, engines are unloaded.
This way, there are no previously filled hash tables messing up things (infinitesimally).
Matthias.
Any chance of this being made an option?
Not in game play.
I may consider the option for testsuites but dread the work involved.
I'm also interested in faster Arena like ChessGUI to test EPD's, Mathias
I agree that ChessGUI is now far better than Arena at awarding points...!
I don't have any problems with regular test suites in Arena. Is there an easy way to convert the STS suites to regular suites, with only a best move, and not the scored moves?
David Dahlem wrote:I don't have any problems with regular test suites in Arena. Is there an easy way to convert the STS suites to regular suites, with only a best move, and not the scored moves?
Good idea!
Arena erroneously includes new move into "searching move" as it doesn't understand partial credit moves.
However, Arena works perfectly when there is only best move in the Epds. (without any secondary moves in Id tag)
I hope someone can come up with best moves only epds of these 900 positions. I could offer this as separate download. I've asked Dann if he could do it with the tools he has.
David Dahlem wrote:I don't have any problems with regular test suites in Arena. Is there an easy way to convert the STS suites to regular suites, with only a best move, and not the scored moves?
Good idea!
Arena erroneously includes new move into "searching move" as it doesn't understand partial credit moves.
However, Arena works perfectly when there is only best move in the Epds. (without any secondary moves in Id tag)
I hope someone can come up with best moves only epds of these 900 positions. I could offer this as separate download. I've asked Dann if he could do it with the tools he has.
It takes 8 minutes to write that program. If nobody else does, I may.
David Dahlem wrote:I don't have any problems with regular test suites in Arena. Is there an easy way to convert the STS suites to regular suites, with only a best move, and not the scored moves?
Good idea!
Arena erroneously includes new move into "searching move" as it doesn't understand partial credit moves.
However, Arena works perfectly when there is only best move in the Epds. (without any secondary moves in Id tag)
I hope someone can come up with best moves only epds of these 900 positions. I could offer this as separate download. I've asked Dann if he could do it with the tools he has.
Another option would be to convert the STS positions to Gradual Test format. This tool is made especially for partial credit scores, and it works perfectly. I can convert the epd positions manually, but not 900 of them, that would take forever.
Here is a sample of Gradual Test format -
1kr5/3n4/q3p2p/p2n2p1/PppB1P2/5BP1/1P2Q2P/3R2K1 w - - bm f5 Bf2 Be5+ Bg4; c1 "10 3 2 2"; id "Undermine.001";
David Dahlem wrote:I don't have any problems with regular test suites in Arena. Is there an easy way to convert the STS suites to regular suites, with only a best move, and not the scored moves?
Good idea!
Arena erroneously includes new move into "searching move" as it doesn't understand partial credit moves.
However, Arena works perfectly when there is only best move in the Epds. (without any secondary moves in Id tag)
I hope someone can come up with best moves only epds of these 900 positions. I could offer this as separate download. I've asked Dann if he could do it with the tools he has.
It takes 8 minutes to write that program. If nobody else does, I may.
It takes only seconds with an editor that has regular expression search and replace. Replace c0.*; with nothing in all the files and you are done.
look here http://www.mediafire.com/file/zzot2ntmnfi/sts.ZIP
David Dahlem wrote:I don't have any problems with regular test suites in Arena. Is there an easy way to convert the STS suites to regular suites, with only a best move, and not the scored moves?
Good idea!
Arena erroneously includes new move into "searching move" as it doesn't understand partial credit moves.
However, Arena works perfectly when there is only best move in the Epds. (without any secondary moves in Id tag)
I hope someone can come up with best moves only epds of these 900 positions. I could offer this as separate download. I've asked Dann if he could do it with the tools he has.
It takes 8 minutes to write that program. If nobody else does, I may.
It takes only seconds with an editor that has regular expression search and replace. Replace c0.*; with nothing in all the files and you are done.
look here http://www.mediafire.com/file/zzot2ntmnfi/sts.ZIP
Thanks!
I will offer this up as separate download with credit to you.
I hope the Arena bug of introducing new move wouldn't be seen when testing these.
