How effective is move ordering from TT?

[lkaufman]

I think any engine that is not within 50 elo of the top is unlikely to be best at anything important.

I don't think it's 50 points, maybe it's 200 points. It's well known that in certain mainstream openings some engines perform better or worse than on average, by at least 50 points. Therefore in some important problems which are more extreme, say fortresses or closed positions or similar, some generally weaker by 200 points engines could perform better. That's why it's a time when having a Junior or a Hiarcs is still not redundant.

Kai

Yes, you are quite right, I should have said "anything of general applicability" or something like that, meaning broad categories like tactics and positional play. I'm pretty sure Shredder is much better than all the top engines at recognizing fortresses, for example.

[syzygy]

But you did jump in as someone else's parrot

???

In case I appeared to be parrotting anyone, it is because I happened to independently have the same view on something.

The reason I jumped in is the discrepancy between "the best", which, at least for me, implies some form of preciseness and presumes the possibility of a meaningful comparison, and the agreement between most here including you and me that a meaningful comparison of the quality of evaluation functions of different engines is not possible.

[Rebel]

I can put this another way. I DO believe our evaluation function is one of the best and possibly the best.

"The best" relative to what... How do you compare the strength of two evaluation functions of two different engines?

Have you not read the posts? I do not believe there is a reasonable test for this.

That is what I read, and I agree.

This is a technical forum, so I assumed that your "belief" was at least a scientific belief.

It's a technical forum but that doesn't mean nobody can make a statements without supplying some sort of formal proof. You are just being silly. It's supposed to be completely open to speculation, assertions, bouncing ideas around and so on.

Again, on a technical forum it seems reasonable that readers assume that such speculations, assertions, bounced ideas are at least testable.

That is not a very scientific or testable statement. "It seems reasonable?" Can you back that up in a way I can test?

I'm being sarcastic to illustrate to you that this is not how the forum operates. You are not even taking your own advice and I have to say that it does NOT seem reasonable to me that you only present ideas that can be tested. That is a sure way to kill innovation and creative thought. Again I have to say that you are just being silly.

I am not saying at all that there is no room for speculation. But you are not speculating, you are expressing non-scientific beliefs.

I have the right to express any belief that I see fit and so do you. Stop trying to become the idea police.

That's fine with me as well, but then don't be surprised that you are being misinterpreted.

Don't worry, I am not bothered by this.

Summary: you have redefined your original claim (Komodo's eval is best) on 2 points. 1) limiting it to non-tactical positions and 2) changed the tone setting, it's no longer a claim but an interesting personal opinion that can not be proven. When you belong to the top people read you and weight every sentence you write. It hasn't been different since day-1.

In my commercial days I made noisy strength statements as well but there are always was some sort of evidence. 2 of such by head:

1. I have stated that Rebel played the best human style of chess. And I offered evidence for that, Rebel scored best overall at AEGON and 3 times was the best computer, no other program achieved that. So there was a base for my claim. And a claim it was.

http://en.wikipedia.org/wiki/Human%E2%8 ... .931997.29

2. In an endless classic discussion about chess knowledge vs search where besides me also Chris held the opinion that an eval can compensate for 1-2 and some more plies I decided put my money where the mouth was and challenged Bob for the NPS match. And while that was only one game due to all the technical problems with auto232, what a game! It demonstrated what was true at that time, a better eval could compensate 4 plies.

Now like you I was a commercial and while such issues are fun and mainly are driven by passion for the game it's impossible to split myself into 2 parts denying there was not some form of commercial interest in the NPS and AEGON examples, of course there was.

If you make a claim you back it up even if perfect conditions are not possible. But I will give you that Vincent quite spoiled the fun with his last posting.

[Houdini]

I was talking about the basic speed of the program. Komodo is outplaying many very strong programs that are doing significantly more nodes per second. It's hard to come to the conclusion that the evaluation function is "average."

One cannot remove "speed" from the equation when considering the quality of an evaluation function. It's quite easy to come up with evaluation terms that add some Elo when they would have no cost in terms of speed. The whole challenge for engine authors is to add evaluation terms that add more strength than their computational cost.

This is also what makes Vincent's claims useless. He may or may not have a very advanced, super-duper evaluation function with countless terms using complex pattern recognition or neural networks. But if he wins 25 Elo by losing 90% of the speed, it's not a good evaluation function.

Robert

[diep]

I was talking about the basic speed of the program. Komodo is outplaying many very strong programs that are doing significantly more nodes per second. It's hard to come to the conclusion that the evaluation function is "average."

One cannot remove "speed" from the equation when considering the quality of an evaluation function. It's quite easy to come up with evaluation terms that add some Elo when they would have no cost in terms of speed. The whole challenge for engine authors is to add evaluation terms that add more strength than their computational cost.

This is also what makes Vincent's claims useless. He may or may not have a very advanced, super-duper evaluation function with countless terms using complex pattern recognition or neural networks. But if he wins 25 Elo by losing 90% of the speed, it's not a good evaluation function.

Robert

Look we don't disagree that sometimes speed is relevant.

But if you just do claims about evaluation function then you can measure how good your evaluation function is and what is the best evaluation function.

Ed proposed a few tests to measure it and Komodo team declined both proposals.

Now Ed doesn't know much from parameter tuning otherwise he would easily know how to measure an evaluation functions quality.

Yet you must take his proposals serious.

If you say that ELOWISE the speed of an evaluation function matters as well in your world, i won't argue there.

Speed is an easy way of obtaining points, especially in rapid time controls of course.

That is however a total different discussion and as soon as you factor in speed you also must factor in the total Ghzminute a move you have and how selective one searches and learning and other tricks.

That's what we call elo.

That's a total different discussion from claiming you have worlds most sophisticated evaluation function whereas obviously Komodo is very similar to rybka's evaluation function with a tad extra passed pawn knowledge, what DeepSjeng already had years before that - and that isn't public versiosn of DeepSjeng, yet there is a clustersjeng online you can play with komodo to see it and it kibitzes evaluations so you can compare ok.

Komodo doesn't have that much extra you know. It's similar to DeepSjeng in terms of EVALUATION.

So basically they claim having a strong evaluation function, yet 99% of its strength they didn't do much effort for. An average guy with some woodcrafts skills can also build a chair if i show him exactly how it must look like. In fact i put in his room the chair i want. "build this". Then with admitted debugging skills of others it isn't that hard you know.

Then put a licensed chairsitter in the team and brag loud you designed it entirely yourself.

And there is a couple of dozens of engines out there now that basically all have a similar evaluation function now. Most of course copy it directly like you did, others add a few things (or try).

Only some of the old programs that took decades to produce that didn't show up suddenly at elo 3000, they programmed every elopoint themselves.

It's easy to claim of course you designed the space shuttle if all you did was give it a new paint (and even giving it a new paint isn't easy as the paint may not damage the shuttle itself when it burns off).

If you take over things from others with a similar engine you don't have to tune that pattern, you already know more or less it works and what it roughly has to be tuned. So there is no tuningswork involved then.

As with playing games you can't tune that parameter of course as it would require millions of games.

Crafty is doing 40k games a test and see how bad Crafty has been tuned and crafty has been doing this at 100-200 cores or so? Who else has 200 cores?

Just playing games doesn't tune for you of course.

Vincent

[Uri Blass]

I do not think that Crafty is bad tuned and my guess is that the main reason that Crafty is weaker is inferior search and not inferior evaluation.

I did not look at Crafty's code but I read that
Crafty does not use LMP(late move pruning when the remaining depth is small) and only use LMR unlike other top programs like stockfish that use both of them.

[Ajedrecista]

Hello again:

I am using the InBetween interface between the engines and LB. Google and download InBetween. Setting these options in the InBetween.ini, which must be in the same folder with InBetween.exe and the engine

Code: Select all

CommandLine := Critter_1.4_64bit.exe 
go movetime 100 := go nodes 200

takes the engine Critter 1.4 from calculating for 100ms move to calculating 200 nodes move. When you set in LB fixed time/move to 100ms, it transforms in fixed nodes at 200. You could as well use

Code: Select all

go movetime 100:= go depth 1

this will take fixed time/move in LB 100ms to depth 1 instead. It depends on the engine which UCI commands it understands, to say, I cannot have Houdini 1.5 and Rybka 4 at fixed nodes.

For example, here is an example. Correcting for engine's speed on my comp, I take fixed nodes/move for engines as:

Critter 1.4 200 nodes/move
Komodo 4 110 nodes/move
StockFish 2.2 140 nodes/move
Rybka 3 10 nodes/move

These are adjusted for speed of engines on my comp, so that the time taken is similar for all engines. I get this

Code: Select all

 

Games Completed = 1200 of 10000 (Avg game length = 1.750 sec) 
Settings = RR/16MB/100ms per move/M 400000cp for 1000 moves, D 120000 moves/PGN:C:\Users\Ani\Downloads\LittleBlitzer\swcr.pgn(5120) 
Time = 2658 sec elapsed, 19492 sec remaining 
 1.  Komodo 4                    254.5/600   215-306-79     (L: m=306 t=0 i=0 a=0)   (D: r=31 i=26 f=15 s=7 a=0)   (tpm=29.9 d=2.21 nps=1457556) 
 2.  Critter 1.4                 172.0/600   131-387-82     (L: m=387 t=0 i=0 a=0)   (D: r=37 i=22 f=11 s=12 a=0)   (tpm=12.6 d=3.30 nps=24557) 
 3.  Rybka 3                     341.0/600   293-211-96     (L: m=211 t=0 i=0 a=0)   (D: r=36 i=35 f=11 s=14 a=0)   (tpm=10.8 d=1.47 nps=27803) 
 4.  StockFish 2.2               432.5/600   390-125-85     (L: m=125 t=0 i=0 a=0)   (D: r=40 i=31 f=5 s=9 a=0)   (tpm=11.6 d=4.35 nps=254992) 

This is somehow equivalent to an extremely short time control, like move in 0.1ms or game in 10ms, which is impossible to set as time/move but possible as nodes/move. Critter 1.4 is pretty representative for IvanHoe and Houdini, and I can suspect that they have a lighter eval than Komodo, and that StockFish and Rybka 3 have some solid evals. It is consistent with the claim of Richard Vida that Ippos have lighter evals than Rybka 3.

Kai

Thank you very much for the info. InBetween can be downloaded from here:

http://home.online.no/~malin/sjakk/

I reached that web thanks to this other web (and Google, of course!):

http://www.chessville.com/links/links_comp_computer.htm

I tried exactly what you said, but disgracefully it does not work for me. I want to fix depth for Quazar 0.4 w32 (a strong, single core engine) but I do not manage it... I do the following:

a) I create two separate folders: one for Quazar (depth 1) and the other for Quazar (depth 2). I create an InBetween.ini file in each folder:

Code: Select all

CommandLine := Quazar_0.4_w32.exe
go movetime 100 := go depth 1

Code: Select all

CommandLine := Quazar_0.4_w32.exe
go movetime 100 := go depth 2

I also include one InBetween.exe programme in each folder.

Quazar reaches depth 2 in 0.1 seconds for sure. This engine also recognizes 'go movetime' and 'go depth' commands.

b) I create Engines.lbe file:

Code: Select all

Engine=H:\d1\Quazar_0.4_w32.exe
Hash=16
LB_Name=Depth 1
Engine=H:\d2\Quazar_0.4_w32.exe
Hash=16
LB_Name=Depth 2

c) I create Tournaments.lbt file using LB 2.5 GUI, setting 100 ms for fixed time/move and 16 MB for hash. I start the match but both engines play up to depth 10, 11, 12, 13... and not depth 1 and depth 2.

------------------------

I replaced Quazar_0.4_w32.exe by InBetween.exe in Engines.lbe file, but then LB can not load the engines. Anyway, thanks again for your help.

If anyone knows the solution, he/she will be welcomed! Thanks in advance.

------------------------

Regarding the heaviness of evaluation functions, it is said in Rybka Forum that Rybka 4 and therefore 4.1 have lighter evaluation functions that Rybka 3, so it explains the gain of around 50 or 60 Elo at non-correspondence time controls (looking at IPON, for example). For this reason, in that forum many correspondence chess players prefer engines with a supposedly heavy eval, as Rybka 3 and Zappa Mexico II. I must say that I have no clue in this issue.

Regards from Spain.

Ajedrecista.

[Laskos]

Thank you very much for the info. InBetween can be downloaded from here:

http://home.online.no/~malin/sjakk/

I reached that web thanks to this other web (and Google, of course!):

http://www.chessville.com/links/links_comp_computer.htm

I tried exactly what you said, but disgracefully it does not work for me. I want to fix depth for Quazar 0.4 w32 (a strong, single core engine) but I do not manage it... I do the following:

a) I create two separate folders: one for Quazar (depth 1) and the other for Quazar (depth 2). I create an InBetween.ini file in each folder:

Code: Select all

CommandLine := Quazar_0.4_w32.exe
go movetime 100 := go depth 1

Code: Select all

CommandLine := Quazar_0.4_w32.exe
go movetime 100 := go depth 2

I also include one InBetween.exe programme in each folder.

Quazar reaches depth 2 in 0.1 seconds for sure. This engine also recognizes 'go movetime' and 'go depth' commands.

b) I create Engines.lbe file:

Code: Select all

Engine=H:\d1\Quazar_0.4_w32.exe
Hash=16
LB_Name=Depth 1
Engine=H:\d2\Quazar_0.4_w32.exe
Hash=16
LB_Name=Depth 2

c) I create Tournaments.lbt file using LB 2.5 GUI, setting 100 ms for fixed time/move and 16 MB for hash. I start the match but both engines play up to depth 10, 11, 12, 13... and not depth 1 and depth 2.

------------------------

I replaced Quazar_0.4_w32.exe by InBetween.exe in Engines.lbe file, but then LB can not load the engines. Anyway, thanks again for your help.

If anyone knows the solution, he/she will be welcomed! Thanks in advance.

------------------------

Yes, you must do this step, replace the engines with InBetween.exe in the Engines.lbe file. You can also rename InBetween.exe (but not InBetween.ini) to whatever exe you want to not have confusions in the .lbe file.

Regarding the heaviness of evaluation functions, it is said in Rybka Forum that Rybka 4 and therefore 4.1 have lighter evaluation functions that Rybka 3, so it explains the gain of around 50 or 60 Elo at non-correspondence time controls (looking at IPON, for example). For this reason, in that forum many correspondence chess players prefer engines with a supposedly heavy eval, as Rybka 3 and Zappa Mexico II. I must say that I have no clue in this issue.

Regards from Spain.

Ajedrecista.

[Adam Hair]

Hello again:

I am using the InBetween interface between the engines and LB. Google and download InBetween. Setting these options in the InBetween.ini, which must be in the same folder with InBetween.exe and the engine

Code: Select all

CommandLine := Critter_1.4_64bit.exe 
go movetime 100 := go nodes 200

takes the engine Critter 1.4 from calculating for 100ms move to calculating 200 nodes move. When you set in LB fixed time/move to 100ms, it transforms in fixed nodes at 200. You could as well use

Code: Select all

go movetime 100:= go depth 1

this will take fixed time/move in LB 100ms to depth 1 instead. It depends on the engine which UCI commands it understands, to say, I cannot have Houdini 1.5 and Rybka 4 at fixed nodes.

For example, here is an example. Correcting for engine's speed on my comp, I take fixed nodes/move for engines as:

Critter 1.4 200 nodes/move
Komodo 4 110 nodes/move
StockFish 2.2 140 nodes/move
Rybka 3 10 nodes/move

These are adjusted for speed of engines on my comp, so that the time taken is similar for all engines. I get this

Code: Select all

 

Games Completed = 1200 of 10000 (Avg game length = 1.750 sec) 
Settings = RR/16MB/100ms per move/M 400000cp for 1000 moves, D 120000 moves/PGN:C:\Users\Ani\Downloads\LittleBlitzer\swcr.pgn(5120) 
Time = 2658 sec elapsed, 19492 sec remaining 
 1.  Komodo 4                    254.5/600   215-306-79     (L: m=306 t=0 i=0 a=0)   (D: r=31 i=26 f=15 s=7 a=0)   (tpm=29.9 d=2.21 nps=1457556) 
 2.  Critter 1.4                 172.0/600   131-387-82     (L: m=387 t=0 i=0 a=0)   (D: r=37 i=22 f=11 s=12 a=0)   (tpm=12.6 d=3.30 nps=24557) 
 3.  Rybka 3                     341.0/600   293-211-96     (L: m=211 t=0 i=0 a=0)   (D: r=36 i=35 f=11 s=14 a=0)   (tpm=10.8 d=1.47 nps=27803) 
 4.  StockFish 2.2               432.5/600   390-125-85     (L: m=125 t=0 i=0 a=0)   (D: r=40 i=31 f=5 s=9 a=0)   (tpm=11.6 d=4.35 nps=254992) 

This is somehow equivalent to an extremely short time control, like move in 0.1ms or game in 10ms, which is impossible to set as time/move but possible as nodes/move. Critter 1.4 is pretty representative for IvanHoe and Houdini, and I can suspect that they have a lighter eval than Komodo, and that StockFish and Rybka 3 have some solid evals. It is consistent with the claim of Richard Vida that Ippos have lighter evals than Rybka 3.

Kai

Thank you very much for the info. InBetween can be downloaded from here:

http://home.online.no/~malin/sjakk/

I reached that web thanks to this other web (and Google, of course!):

http://www.chessville.com/links/links_comp_computer.htm

I tried exactly what you said, but disgracefully it does not work for me. I want to fix depth for Quazar 0.4 w32 (a strong, single core engine) but I do not manage it... I do the following:

a) I create two separate folders: one for Quazar (depth 1) and the other for Quazar (depth 2). I create an InBetween.ini file in each folder:

Code: Select all

CommandLine := Quazar_0.4_w32.exe
go movetime 100 := go depth 1

Code: Select all

CommandLine := Quazar_0.4_w32.exe
go movetime 100 := go depth 2

I also include one InBetween.exe programme in each folder.

Quazar reaches depth 2 in 0.1 seconds for sure. This engine also recognizes 'go movetime' and 'go depth' commands.

b) I create Engines.lbe file:

Code: Select all

Engine=H:\d1\Quazar_0.4_w32.exe
Hash=16
LB_Name=Depth 1
Engine=H:\d2\Quazar_0.4_w32.exe
Hash=16
LB_Name=Depth 2

c) I create Tournaments.lbt file using LB 2.5 GUI, setting 100 ms for fixed time/move and 16 MB for hash. I start the match but both engines play up to depth 10, 11, 12, 13... and not depth 1 and depth 2.

------------------------

I replaced Quazar_0.4_w32.exe by InBetween.exe in Engines.lbe file, but then LB can not load the engines. Anyway, thanks again for your help.

If anyone knows the solution, he/she will be welcomed! Thanks in advance.

------------------------

Regarding the heaviness of evaluation functions, it is said in Rybka Forum that Rybka 4 and therefore 4.1 have lighter evaluation functions that Rybka 3, so it explains the gain of around 50 or 60 Elo at non-correspondence time controls (looking at IPON, for example). For this reason, in that forum many correspondence chess players prefer engines with a supposedly heavy eval, as Rybka 3 and Zappa Mexico II. I must say that I have no clue in this issue.

Regards from Spain.

Ajedrecista.

I will try to replicate your problem when I return home in a few hours. As Kai said, LB has to be directed to InBetween. In your first attempt, LB was starting the engines directly, bypassing InBetween and the depth instructions.

Have you created InBetween logfiles to see the communication between LB and InBetween?

[Ajedrecista]

Hi Adam:

I will try to replicate your problem when I return home in a few hours. As Kai said, LB has to be directed to InBetween. In your first attempt, LB was starting the engines directly, bypassing InBetween and the depth instructions.

Have you created InBetween logfiles to see the communication between LB and InBetween?

Thanks for your interest, and also thanks to Kai for the help.

I did not create logfiles; if I tick 'Log First Engine (LittleBlitzer.log)' option in LB 2.5 GUI, I obtain this info:

Code: Select all

-->((null)) uci
<--(&#40;null&#41;) InBetween. 1.4 by Odd Gunnar Malin
<--(&#40;null&#41;) Syntax&#58; InBetween &#91;/t filename /p priority /c /d n /l logfile&#93; <command line>
<--(&#40;null&#41;) Where&#58;
<--(&#40;null&#41;)   /t textfile with the translation. Default=InBetween.ini
<--(&#40;null&#41;)   /p Priority flag, low, normal or high. Default is the same as this process.
<--(&#40;null&#41;)   /c Stop ctrl C and ctrl break. Default is passing through.
<--(&#40;null&#41;)   /d Debug where n could be 1=client side, 2=server side, 3=both.
<--(&#40;null&#41;)   /l Logfile.

But the engines do not load. As a side note, I must say that I have changed a little InBetween.ini files. I now write:

Code: Select all

CommandLine := H:\d1\Quazar_0.4_w32.exe
go movetime 100 := go depth 1
Log := logfile.log

I mean, I write the full path in 'CommandLine'; I also add that last line, although I have not seen any logfile.log file created.

A normal LittleBlitzer.log file, loading two copies of Quazar, looks like this:

Code: Select all

-->((null)) uci
<--(&#40;null&#41;) id name Quazar 0.4 w32
<--&#40;Depth 1&#41; id author Dmitry Morozov
<--&#40;Depth 1&#41; option name Hash type spin default 32 min 4 max 8192
<--&#40;Depth 1&#41; option name Clear Hash type button
<--&#40;Depth 1&#41; option name Ponder type check default true
<--&#40;Depth 1&#41; option name MultiPV type spin default 1 min 1 max 500
<--&#40;Depth 1&#41; uciok
-->&#40;Depth 1&#41; setoption name Hash value 1
-->&#40;Depth 1&#41; setoption name Ponder value false
-->&#40;Depth 1&#41; setoption name OwnBook value false
-->&#40;Depth 1&#41; setoption name Hash value 16
-->&#40;Depth 1&#41; isready
<--&#40;Depth 1&#41; readyok
-->&#40;Depth 1&#41; quit
-->(&#40;null&#41;) uci
<--(&#40;null&#41;) id name Quazar 0.4 w32
<--&#40;Depth 2&#41; id author Dmitry Morozov
<--&#40;Depth 2&#41; option name Hash type spin default 32 min 4 max 8192
<--&#40;Depth 2&#41; option name Clear Hash type button
<--&#40;Depth 2&#41; option name Ponder type check default true
<--&#40;Depth 2&#41; option name MultiPV type spin default 1 min 1 max 500
<--&#40;Depth 2&#41; uciok
-->&#40;Depth 2&#41; setoption name Hash value 1
-->&#40;Depth 2&#41; setoption name Ponder value false
-->&#40;Depth 2&#41; setoption name OwnBook value false
-->&#40;Depth 2&#41; setoption name Hash value 16
-->&#40;Depth 2&#41; isready
<--&#40;Depth 2&#41; readyok
-->&#40;Depth 2&#41; quit

I want to do a small experiment of diminishing returns between Elo gain and search depth, maybe with 2500 games/match, without opening books, EPD files, etc. The idea is let play (depth 1) vs. (depth 2), (depth 2) vs. (depth 3), ..., (depth N) vs. [depth (N - 1)]. I know that there are several attempts:

http://www.talkchess.com/forum/viewtopi ... =0&t=29011

http://www.top-5000.nl/ply.htm

http://rybkaforum.net/cgi-bin/rybkaforu ... ?tid=18656

Just mentioning a few of them; CPW page dedicated to 'depth' can be of interest.

In the case that I finally manage to run fixed depth matches, then probably I will get interesting results and will post them in this subforum; if not (the most probable case), all remains the same and there is no problem for me. I know that the best thing is to compare various engines, but I have not got time and resources to do this, so I only randomly choose Quazar because it is an engine I like.

I think that cutechess-cli can run fixed depth matches, although it is very complicated to me its use (please remember than I am almost a newbie to computers), so I would prefer LB if possible (at least, I more less know how to use LB).

Regards from Spain.

Ajedrecista.