Don,My intuition is the same as youDon wrote:Intuition is misleading, you are basing this all on your intuition.Uri Blass wrote:If you turn off part of the evaluation that is relevant for all the game then I agree but it may be different for adding some complicated evaluation that is about winning won endgame.
I can imagine that removing knowledge of how to win won endgames can scale well because at fast time control removing the knowledge may prevent the program to win won endgame when at slow time control there is no demage because the program win the endgame by search after it get it.
In fact I have the opposite intuition, that if you don't know which ending is a win, loss or draw it is too late to figure it out once you are in it. So my guess here is that knowledge of endings and whether they are won or drawn is more important for a deep searchers.
However, my intuition may be wrong too. What matters is reality and the actual studies done on this and it's pretty clear that knowledge is scalable as a general rule.
I do think that certain knowledge may not be scalable and may work especially well on short searches.
The kind of knowledge you are talking about here is not particularly relevant, the most important knowledge is based on evaluating a position correctly so that when you make decisions you go into the right sort of position - the kind that makes a difference between win and draw and lose. Knowing how to win a won game is useful for very shallow searches but this is not the kind of knowledge that is scalable. It's technique knowledge, not deep knowledge.
For example it may be possible that without tablebases specific knowledge how to win endgames like KBN vs K may be productive at fast time control when this knowledge does not help at slow time control because the program can find the win by search.
Komodo actually has specific knowledge on how to win this ending. Not because it really helps but because all my programs have had this and it's almost like something inherited. Like you say it can win this without the knowledge, but on a hyper blitz level it is possible that it will make a difference. But that is not scalable knowledge.
I have related this story on the forum before so I will keep this short - but it did remind me of something I already know.
More general case is simply replacing every evaluation that is bigger than n pawns and is not mate by n pawns for some big n.
You can describe it as removing knowledge and it may cause big problems at fast time control because the program will not be able to win some won endgame but cause no problem at slow time control because the program is going to find the mate by search.
Uri
In the early days of Doch I was using Glaurung as a sparring partner and trying to "catch" it. Doch was getting stronger every day and at one point I announced to Larry that we had finally caught and passed Glaurung. I was basing this on the hyper blitz time control tests were were doing, such as 3 seconds per game. I quickly discovered that Glaurung still owned us when testing at more realistic time controls and even did a graph which showed that Doch was stronger at fast time controls and weaker at longer ones. The lines were not parallel at all, in fact the difference was extreme once you got to 1 minute - something like 150 ELO or more. It was ridiculous.
Larry and I wasted a lot of time on this, I think it was many weeks trying to determine what the scalability problem was. Clearly, there was something wrong with Doch and we tried every kind of test and modification we could imagine. Were we too selective? Not selective enough? Was it futility pruning or our forward pruning tricks? Was null move pruning broken?
Eventually, I got frustrated at the lack of progress and told Larry we needed to move on for a while and come back to this later. So we began working on low hanging fruit, the weakest part of Doch at the time, king safety. Doch responded enormously to our fixing this problem and was suddenly much stronger - but to both our surprises the scalability issue completely went away! We were now beating this version of Glaurung at every level!
Even though I already knew this, it was a reminder about how scalability works. No matter what depth you search, you are going to encounter end nodes that you do not understand. You MUST be able to tell which is better or else adding even 20 ply to the search may not matter. However know that some configuration of KBNvsK is better than another is not going to make the slightest bit of difference unless you are already in that ending. At best it will tell the program how to get into a version of it that wins a little quicker - not really useful for ELO points. But your king can be in danger for 10, 20 or more moves and if your program doesn't understand that it will make mistakes constantly, game losing mistakes that 10 extra ply won't be able to rectify. THAT is why Doch was not scaling, because the extra depth helped Doch tactically and in many other ways, but it was still vulnerable to anything related to king safety - it was like the extra depth was close to meaningless.
And that is what you will get with a lack of knowledge, big holes in the program that you cannot brute force your way out of. Each extra ply of search will cause your program to fall a little more behind a program that understands these things even at end nodes.
I think you are guilty of thinking about the root position and what a program can do - can it win this ending, can it do this, can it do that. But what really matters is what it understands at END node evaluation time. If it understands something in the evaluation then every extra ply of search makes it stronger - it can see that thing 1 ply deeper. But if it doesn't understand it then each additional ply has no impact - at least until it is looking so deeply that it can solve it tactically. Sometimes you can survive a king safety attack for 20 or 30 ply before finally losing material or getting mated - so not having king safety makes your program 20 or 30 ply weaker in many positions.
I did not claim that knowing if an endgame is winning or draw is less important at long time control.
I claimed only that knowing to win won endgame is more important at fast time control(and it is clearly knowledge to know if to evaluate position as +5 or +6 or +7.
Reading that adding king safety helped komodo to score better at longer time control does not surprise me.
Note that I suspect that reducing the maximal evaluation that is not mate to +5 pawns may scale well because at long time control the program is going to have no problem to win won position and the program may also get more positions in the hash table for the same size of hash tables because it is going to need less space to store the evaluation number(it may be possible to reduce the mate value).
I consider reducing the maximal evaluation as clearly removing knowledge from the program and I think that rybka has some similiar bug when she scored evaluation of more than +5.12 as +5.12.
