So now that I've got a relatively stable engine, I'd like to do some quick and dirty testing against other engines, perhaps using the nunn positions.
There's a bushel of strong engines that support setboard, but I'm having a bit of trouble locating weak ones. (well, weak ones other than my own, anyway...)
Do all y'all have any suggestions? I appreciate it!
Edit: Also, I have gradualtest.exe. Is there anything significantly better for running an EPD suite?
weaker winboard engines that support setboard?
Moderators: hgm, Rebel, chrisw
-
- Posts: 718
- Joined: Fri Mar 20, 2009 8:59 pm
-
- Posts: 558
- Joined: Sat Mar 25, 2006 8:27 pm
Re: weaker winboard engines that support setboard?
Beaches supports setboard. It's in the same ballpark as TSCP.
-
- Posts: 718
- Joined: Fri Mar 20, 2009 8:59 pm
Re: weaker winboard engines that support setboard?
Excellent, thanks for the tip!
Well, olithink and beaches certainly bracket my engine, though probably too widely to be terribly useful...
Well, olithink and beaches certainly bracket my engine, though probably too widely to be terribly useful...
Code: Select all
Rank Name Elo + - games score oppo. draws
1 OliThink 5.1.8alpha 374 135 98 20 88% 69 5%
2 Moneypenny 1.5 69 90 97 40 56% -35 3%
3 Beaches 2.26 -444 133 283 20 0% 69 0%
-
- Posts: 6640
- Joined: Thu Mar 09, 2006 4:21 am
Re: weaker winboard engines that support setboard?
Hi Matt, I'd like to see how your engine performs in this Undermining test suite, If possible, can you please run this test on perhaps 2 minutes per position and let me know of the results?MattieShoes wrote:So now that I've got a relatively stable engine, I'd like to do some quick and dirty testing against other engines, perhaps using the nunn positions.
There's a bushel of strong engines that support setboard, but I'm having a bit of trouble locating weak ones. (well, weak ones other than my own, anyway...)
Do all y'all have any suggestions? I appreciate it!
Edit: Also, I have gradualtest.exe. Is there anything significantly better for running an EPD suite?
http://computerchessblogger.googlepages.com/sts
I have sample test scores from wide range of engines, so perhaps I might have an idea of how your engine ranks up in positional understanding.
Next test suite on Open Files and Diagonals is getting finished, Dann Corbit will finalise the select 100 set before release.
PS: Arena 1.99 beta 5 or the latest one has an automatic test feature where you can use it to test your engine on EPDs on given time, and several other conditions.
-
- Posts: 6401
- Joined: Thu Mar 09, 2006 8:30 pm
- Location: Chicago, Illinois, USA
Re: weaker winboard engines that support setboard?
You do not need that. Winboard can load positions in pgn and nunn positions are available in pgn too. Besides, pgn is preferable because it avoids weird situations with time management (in some engines). BTW, I do not recommend to use the nunn positions. Use them in combination with many others such as silver and noomen. It is not a good idea to use a very small set of positions.MattieShoes wrote:So now that I've got a relatively stable engine, I'd like to do some quick and dirty testing against other engines, perhaps using the nunn positions.
There's a bushel of strong engines that support setboard, but I'm having a bit of trouble locating weak ones. (well, weak ones other than my own, anyway...)
Do all y'all have any suggestions? I appreciate it!
Edit: Also, I have gradualtest.exe. Is there anything significantly better for running an EPD suite?
I have try them (I will), but maybe you would be interested in this set (EasyWay)
http://wbec-ridderkerk.nl/html/details1/RomiChess.html
I am collecting my own, handpicking them from ECO, but it will take a while (a long while)
Miguel
Edit: Gaviota supports setboard.
-
- Posts: 718
- Joined: Fri Mar 20, 2009 8:59 pm
Re: weaker winboard engines that support setboard?
I'd be happy to
I noticed the EPD has only one best move listed, but it has several listed in a comment. As far as I can tell, Arena or GradualTest will both only give credit if it's listed in the bm tag. Gradualtest has its own odd system for weighted best moves like Tonypos.epd that looks like this:
c1 contains the "scores" for each move.
Were you looking for just the # of best moves or some sort of scored result like in tonypos?
Matt
Edit: I ran it 10 seconds/position just to see if it works okay with Arena and it got 50/100. I'll run a longer test in a bit.
I noticed the EPD has only one best move listed, but it has several listed in a comment. As far as I can tell, Arena or GradualTest will both only give credit if it's listed in the bm tag. Gradualtest has its own odd system for weighted best moves like Tonypos.epd that looks like this:
Code: Select all
1r2r1k1/1pqbbppp/p2p1n2/4p3/P3PP2/2N1BB2/1PP2QPP/R4R1K b - - bm Rbc8 exf4 Bc6 Rec8 b6 b5 Be6 h6; c0 "Karpov-Kasparov, Moscow 1985"; c1 "10 9 7 6 6 5 4 3"; id "T01";
Were you looking for just the # of best moves or some sort of scored result like in tonypos?
Matt
Edit: I ran it 10 seconds/position just to see if it works okay with Arena and it got 50/100. I'll run a longer test in a bit.
Re: weaker winboard engines that support setboard?
Hi,
my engine frank-walter should be in that ball park, and I even have several versions available (probably in the range between beeches and olithink^^)
http://chess.winkelhagen.com
The only thing it that it's a java engine, so some additional set-up might be required.
my engine frank-walter should be in that ball park, and I even have several versions available (probably in the range between beeches and olithink^^)
http://chess.winkelhagen.com
The only thing it that it's a java engine, so some additional set-up might be required.
-
- Posts: 6640
- Joined: Thu Mar 09, 2006 4:21 am
Re: weaker winboard engines that support setboard?
Hi Matt, I was mainly looking for just the # of best moves. Scores given for other moves as in Tony's test evaluation was implemented by Dann Corbit. Perhaps he knows how to run those kind of test and get the scores based on guesses order for each engines, I haven't tried that yet. I'd be happy if someone else explains to me how to do just that in Arena.MattieShoes wrote:I'd be happy to
I noticed the EPD has only one best move listed, but it has several listed in a comment. As far as I can tell, Arena or GradualTest will both only give credit if it's listed in the bm tag. Gradualtest has its own odd system for weighted best moves like Tonypos.epd that looks like this:c1 contains the "scores" for each move.Code: Select all
1r2r1k1/1pqbbppp/p2p1n2/4p3/P3PP2/2N1BB2/1PP2QPP/R4R1K b - - bm Rbc8 exf4 Bc6 Rec8 b6 b5 Be6 h6; c0 "Karpov-Kasparov, Moscow 1985"; c1 "10 9 7 6 6 5 4 3"; id "T01";
Were you looking for just the # of best moves or some sort of scored result like in tonypos?
Matt
Edit: I ran it 10 seconds/position just to see if it works okay with Arena and it got 50/100. I'll run a longer test in a bit.
50/100 in 10 seconds per position is not bad, Rotor scored 70 something in 7 minutes per position. So I guess your engine could get past 60's when given more time.
-
- Posts: 27811
- Joined: Fri Mar 10, 2006 10:06 am
- Location: Amsterdam
- Full name: H G Muller
Re: weaker winboard engines that support setboard?
The various versions of micro-Max all support setting up of a position, although they don't do it through the setboard command but through the WB protocol v1 edit command. And they span Elo ranges from ~1500 (uMax 1.6) to 2050 (uMax 4.8_PII).
Beware that many of the weaker engines also behave weird w.r.t. time control after loading a position as PGN (not counting the moves that were forced). Such engines would only work well with incremental / sudden-death time controls when starting from a loaded game.
Beware that many of the weaker engines also behave weird w.r.t. time control after loading a position as PGN (not counting the moves that were forced). Such engines would only work well with incremental / sudden-death time controls when starting from a loaded game.
-
- Posts: 718
- Joined: Fri Mar 20, 2009 8:59 pm
Re: weaker winboard engines that support setboard?
I don't know that it's possible in Arena. I wrote an ugly script in a couple minutes to convert the epd file to what Gradualtest.exe uses for scored moves. The results at 10 sec/position are slightly different (!?) Maybe caused by it bouncing between two PV's and a slight timing difference.
I'll run it at 2 minutes/position now but I suspect it won't do any better. I wouldn't be terribly surprised if it did worse
Code: Select all
Number of tests: 100
Max score: 1000
Engines score: 621
Number with full score: 52
Found in: 2 sec = 1, 5 sec = 15, 10 sec = 36,
Time used for tests (wrong answer=10s): 802s