Question to Larry Kaufman about Rybka

[BubbaTough]

I don't believe the issue is raw speed. Actually Komodo is pretty fast going by NPS (if these figures are really comparable among engines); only Fritz comes to mind as clearly faster. The issue is how Robbolito manages to reach higher search depths than other engines while still remaining competitive at the same depth of search. How do they have their cake and eat it too?

Ahh, gotcha. So in your earlier post when you said:

So if you used only very cheap, simple eval in stockfish do you think you could come close to Robo-speed? If so you would then presumably only lose to Robo because you need riskier pruning to achieve that speed, is that right?

you meant something other than NPS.

So it sounds like you want someone else to carefully comb through the robo code, find the one great secret that makes it so good, and tell you the amazing, simple, and easy to implement idea that will strength your program by 300 elo. Well, perhaps you should ask Don to do that, and tell us the results .

-Sam

[lkaufman]

Well, that would be nice, but most of the programmers who have looked at Robo seem to agree that there is no simple secret, it's just a lot of little things. That is pretty much Don's opinion too, though he hasn't spent much time looking at Robo. But I already know enough about the eval in Rybka to say that a greatly simplified eval as in Robo is not anything special, so it comes down to the search. I also know that the pruning in Robbo, Stockfish, and Komodo are not so drastically different; I mean each does things a bit differently so there are different tradeoffs between speed (as measured in time to iteration) and quality (as measured by results at a given depth if we assume similar eval, which of course is not so easy to do in a case like Stockfish), but no twiddling with parameters or details will make one program a hundred Elo stronger. I guess I'm expecting someone to say something like this: "Robbo achieves its speed by doing XXX. We can't do the same without starting over from scratch and copycatting Robbo." Or "We can't do the same with a complex eval like we have" or "the Robbo speed is achieved at the cost of X, which makes it no good at long time controls" or something like this. In short, why are there no programs that search like Robbo but have totally unrelated evaluation functions?

[BubbaTough]

In short, why are there no programs that search like Robbo but have totally unrelated evaluation functions?

Ahh, well that I can answer, or at least posit an educated guess. Like Don, I looked at Robbo search briefly. For me anyway it was very very hard to understand. Pieces of it are scattered in lots of different files, varibale names are odd, and the decompiled nature adds its own bizarre confusion as well. It would take a lot of time and effort to unravel what they are doing. I would guess most part-time programmers who are just doing this for fun (which is practically everyone on this forum) just cannot be bothered to spend the very significant amount of time necessary to unravel it. I would guess ay few professionals might be willing, but I doubt they will be talking about their conclusions. Most "professionals" in this genre actually have other jobs that they make their income from anyway, and chess programming is still really just a hobby for them so many of them might not delve that far into it either.

It sounds like the Stockfish team may be putting in some effort into this, so I would guess that in the next year or so some of the best nuggets may show up there if they have not already (appropriately credited of course) after which more people will start using searches more similar to Robo since Stockfish is actually coded legibly. Of course, there may be a large number of little tricks that work for Robo and not for Stockfish, and those will stay hidden from all but the most dedicated code spelunkers. If those tricks are numerous enough, robo will remain significantly different in its search from others.

Remember, fruit had a large effect not because it was strong, but because it was SIMPLE and strong. Robo is not, and thus will not have the same kind of influence on people's codebase fruit had.

-Sam

[mcostalba]

I also know that the pruning in Robbo, Stockfish, and Komodo are not so drastically different;

Well, perhaps Komodo is similar to Stockfish, but I can assure you that search in Robbo is very different from SF. There are a good number of fundamental differences that I won't enumerate (in case ask Don that has for sure already done this for himself) but any average programmer that has spent few hours on Robbo could see them without a big effort.

but no twiddling with parameters or details will make one program a hundred Elo stronger.

I don't agree here: Sf 1.5.1 is more then 100 ELO stronger then Glaurung just by means of with what you call " twiddling with parameters or details". The difference between a strong and a weak engine is at 90% _only_ in the details, but in a chess engine there are hundreds, thousands of details that at the end of the day make a big difference. I can state this with confidence because I have seen it directly with the Galurung <-> Stockfish case.

In short, why are there no programs that search like Robbo but have totally unrelated evaluation functions?

In chess engines copy and paste does not work. Robbo is _so_ different from the open sources engines known up to few months ago that it will take years, not months, to completely assimilate the underlying ideas by this community in what will be considered common knwoledge, for instance what today could be null move search or king safety.

[lkaufman]

Thanks for a clear answer, which sounds correct to me. So basically only a few people (at most) understand why the Robbo search is so fast and strong. Do you think even the authors of Robbo and other Rybka derivatives themselves understand the answer to this question?

[mcostalba]

It would take a lot of time and effort to unravel what they are doing.

I sugegst to look at ippolit big ipp_eng.c single file, with a good C++ IDE is not so difficult to browse and a lot of code is redundant becasue is duplicated for white/black and for the different searches, so once you do the effort for one kind of search and one color you have almost done it, remaining code is almost the same with little variations.

[Uri Blass]

I don't believe the issue is raw speed. Actually Komodo is pretty fast going by NPS (if these figures are really comparable among engines); only Fritz comes to mind as clearly faster. The issue is how Robbolito manages to reach higher search depths than other engines while still remaining competitive at the same depth of search. How do they have their cake and eat it too?

I tested komdo against robbolito at fixed depth of 1 ply and 2 plies and it seems that robbolito is clearly stronger at small fixed depth.

I think that you should ask the question why komodo is weaker.
I think that komodo tends to prune too much.

Here is one example.

[D]1r2r1k1/nbpqbp2/5npp/p2Pp3/P1B4N/2PPN2P/2Q2PP1/1RB1R1K1 w - - 0 24

Robbolito can see the tactics Nxg6 at depth 1 when komodo needs clearly bigger depth.

another example

Komodo at depth 2 went into a fork and played Rd5(I understand the reason for it but I think that the rule not to allow bad captures in the qsearch (after Rd5 Be4+ Kf1) is bad and it may be better to allow bad captures at least in cases that the bad capture is a check and another piece is hanging.

[D]8/5pp1/7p/4Pb2/3R1Pk1/6P1/4rBK1/8 w - - 0 45


third example
The last move of komodo was h5 it seems that komodo does not see pins in the qsearch and even after h5 it needs 3 plies to see that

axb3 axb3 Rxb3 is good for black(Rxb3 is considered in the qsearch as a bad capture so it is not considered there and robbolito can find axb3 with positive score for black even at depth 1
[D]1rb1k2r/5pp1/2p1pb1p/2Q4P/p3qP2/1P2B3/P1P3P1/1K1R1B1R b k - 0 19


I believe that komodo can earn from trying to fix the search in order not to miss tactics at small depths.


Edit:In example 2 it seems that robbolito has the same problem and it is only lucky not to play Rd5 but in examples 1 and 3 robbolito seems to have a better search at very small depths.

Uri

[mcostalba]

Do you think even the authors of Robbo and other Rybka derivatives themselves understand the answer to this question?

Probably you know better then me: if Robbo is enough different from Rybka it means they have understood.

[lkaufman]

Depends on what you call "enough". The eval is greatly simplified, and some search parameters are changed, but is that enough to say that they really understand the program? My own feeling is no.

[Milos]

Depends on what you call "enough". The eval is greatly simplified, and some search parameters are changed, but is that enough to say that they really understand the program? My own feeling is no.

A similar question for you. Do you Larry understand Rybka's search and why is it so efficient?