Komodo 5 release now available!

[Graham Banks]

A description of the starting positions Tom is using can be found here.

OK, now we're getting somewhere! That book is only 4 moves (8 ply) deep. I think the testing organizations test with books considerably deeper than this.
I have believed for some time that Komodo was the best engine in the opening phase of the game. If much of the opening is played by the book, this advantage loses its value. So I expected that the deeper the books used for testing, the worse Komodo would look relative to other top engines. Now there is solid evidence to substantiate this.
So it seems that with minimal book help, Komodo is fully competitive with Houdini 2 or even superior, even at fast blitz levels, but when the opening is played by the book Houdini 2 still has the lead. Of course this is not yet proven, as their are other variables in the various tests, such as ponder, type of time control, AMD vs Intel, and average time per game. So we need more data, but at least now we know what we are looking for.
Any info about average number of ply in book for each testing group would be helpful here.

CCRL uses a limit of 12 moves from the opening book, but most of us use less than that.
For example, I use 6 or 8 moves mostly.

[Don]

A description of the starting positions Tom is using can be found here.

OK, now we're getting somewhere! That book is only 4 moves (8 ply) deep. I think the testing organizations test with books considerably deeper than this.
I have believed for some time that Komodo was the best engine in the opening phase of the game. If much of the opening is played by the book, this advantage loses its value. So I expected that the deeper the books used for testing, the worse Komodo would look relative to other top engines. Now there is solid evidence to substantiate this.
So it seems that with minimal book help, Komodo is fully competitive with Houdini 2 or even superior, even at fast blitz levels, but when the opening is played by the book Houdini 2 still has the lead. Of course this is not yet proven, as their are other variables in the various tests, such as ponder, type of time control, AMD vs Intel, and average time per game. So we need more data, but at least now we know what we are looking for.
Any info about average number of ply in book for each testing group would be helpful here.

CCRL uses a limit of 12 moves from the opening book, but most of us use less than that.
For example, I use 6 or 8 moves mostly.

I am very much interested in the variety too. Are the lines in the books that you guy use really solid mainlines and limited to just the top lines or does it have a mixture of gambits and offbeat lines? Are there a LOT of openings or just a couple of hundred or less? Or is it up to each tester?

Thanks,

Don

[Graham Banks]

A description of the starting positions Tom is using can be found here.

OK, now we're getting somewhere! That book is only 4 moves (8 ply) deep. I think the testing organizations test with books considerably deeper than this.
I have believed for some time that Komodo was the best engine in the opening phase of the game. If much of the opening is played by the book, this advantage loses its value. So I expected that the deeper the books used for testing, the worse Komodo would look relative to other top engines. Now there is solid evidence to substantiate this.
So it seems that with minimal book help, Komodo is fully competitive with Houdini 2 or even superior, even at fast blitz levels, but when the opening is played by the book Houdini 2 still has the lead. Of course this is not yet proven, as their are other variables in the various tests, such as ponder, type of time control, AMD vs Intel, and average time per game. So we need more data, but at least now we know what we are looking for.
Any info about average number of ply in book for each testing group would be helpful here.

CCRL uses a limit of 12 moves from the opening book, but most of us use less than that.
For example, I use 6 or 8 moves mostly.

I am very much interested in the variety too. Are the lines in the books that you guy use really solid mainlines and limited to just the top lines or does it have a mixture of gambits and offbeat lines? Are there a LOT of openings or just a couple of hundred or less? Or is it up to each tester?

Thanks,

Don

The choice of books or opening positions is the choice of the tester, but there is plenty of variety.

Take a look at this link to see all of the openings played by ECO classification.
http://www.computerchess.org.uk/ccrl/40 ... y_eco.html

This link shows all of the different opening books used.
http://www.computerchess.org.uk/ccrl/40 ... _book.html

Graham.

[lkaufman]

Would you guess that the average would be about 8 moves?

[Graham Banks]

Would you guess that the average would be about 8 moves?

I would imagine so.

[MM]

A description of the starting positions Tom is using can be found here.

OK, now we're getting somewhere! That book is only 4 moves (8 ply) deep. I think the testing organizations test with books considerably deeper than this.
I have believed for some time that Komodo was the best engine in the opening phase of the game. If much of the opening is played by the book, this advantage loses its value. So I expected that the deeper the books used for testing, the worse Komodo would look relative to other top engines. Now there is solid evidence to substantiate this.
So it seems that with minimal book help, Komodo is fully competitive with Houdini 2 or even superior, even at fast blitz levels, but when the opening is played by the book Houdini 2 still has the lead. Of course this is not yet proven, as their are other variables in the various tests, such as ponder, type of time control, AMD vs Intel, and average time per game. So we need more data, but at least now we know what we are looking for.
Any info about average number of ply in book for each testing group would be helpful here.

I can only say that in blitz, even in fast blitz (e.g. 3'+0'') with opening book of 2 half moves i have Komodo better than Houdini. Anyone who has a decent knowledge of chess and looks at Houdini sees that it plays the opening bad with some antipositional moves (propably coming from ippo-robbo style).
Komodo plays differently althought i think it gives too much space to its opponent and prefers the position instead the tempos.

Larry, you must excuse me if i tell you again, but supporting chess960 you would have an exact estimation of the strenght of Komodo without opening book.

Best Regards

[peter]

Larry, you must excuse me if i tell you again, but supporting chess960 you would have an exact estimation of the strenght of Komodo without opening book.

Maurizio, excuse me too, if I tell you and anybody else who want's to hear it one more time from me and to anybody who does not:
Without book from the one and only normal starting position you have an estimation of the strenght of the engine you test as exact as the rest of your testing provisos are, for example the number of games as for pure engine- engine- matches.
The strength of the engine to deal with the one and only starting position.

Every kind of tournament you let play is some kind of thematic tournament, and the strength of the engines tested depend on the starting positions.
You can do whatever you want to avoid this, you won't succeed.
You can take short books or deep books, selected starting positions of whatever number you can deal with.
If you choose positions similiar to each other you choose certain engines' strengthes, if you choose rather different ones it's a quantitative difference of more or less counting, not a qualitative one.
Playing strength of an engine as well as that of human players depend on the positions on the board.
You can not give any measure of a man's or an engine's strength in playing chess than position-depending ones.
That's the reason, why Elo, as the one and only accepted measure for the strength of men and engines in playing chess, lack more and more, as for engines of too little differences in strength, as for certain kind of positions, especially the ones that are commonly used for testing them.
If you have to have more criterion differences not to need even bigger and bigger samples of data for statistics of some power, as you need now to differentiate similiar engines of similiar strength, you have to have more differences in the positions to test with.
Your idea about Chess960 isn't a bad one, it just hasn't any more meaning than any other kind of certain positional testing, we were doing in ancient times. That's what you can do to test engines' strength as for certain postions of selective interest with well selected positions.
Make thematic matches or thematic positional tests and make yourself conscious that any other kind of testing is the same in kernel, testing of certain positional strengthes only.
Forget the Elo, that's a thing of yesterday as for engines, you need not compare them to humans any more so you should better stop trying to do so and to do as if, just to play act hoping more Elo would sell any better, these days were gone at the latest from the arising of freeware as well as the commercial one as for Elo, that was the rising of Elosion and Celolitis and the end of comparing engines just in this one and only way.

[MM]

Larry, you must excuse me if i tell you again, but supporting chess960 you would have an exact estimation of the strenght of Komodo without opening book.

Maurizio, excuse me too, if I tell you and anybody else who want's to hear it one more time from me and to anybody who does not:
Without book from the one and only normal starting position you have an estimation of the strenght of the engine you test as exact as the rest of your testing provisos are, for example the number of games as for pure engine- engine- matches.
The strength of the engine to deal with the one and only starting position.

Every kind of tournament you let play is some kind of thematic tournament, and the strength of the engines tested depend on the starting positions.
You can do whatever you want to avoid this, you won't succeed.
You can take short books or deep books, selected starting positions of whatever number you can deal with.
If you choose positions similiar to each other you choose certain engines' strengthes, if you choose rather different ones it's a quantitative difference of more or less counting, not a qualitative one.
Playing strength of an engine as well as that of human players depend on the positions on the board.
You can not give any measure of a man's or an engine's strength in playing chess than position-depending ones.
That's the reason, why Elo, as the one and only accepted measure for the strength of men and engines in playing chess, lack more and more, as for engines of too little differences in strength, as for certain kind of positions, especially the ones that are commonly used for testing them.
If you have to have more criterion differences not to need even bigger and bigger samples of data for statistics of some power, as you need now to differentiate similiar engines of similiar strength, you have to have more differences in the positions to test with.
Your idea about Chess960 isn't a bad one, it just hasn't any more meaning than any other kind of certain positional testing, we were doing in ancient times. That's what you can do to test engines' strength as for certain postions of selective interest with well selected positions.
Make thematic matches or thematic positional tests and make yourself conscious that any other kind of testing is the same in kernel, testing of certain positional strengthes only.
Forget the Elo, that's a thing of yesterday as for engines, you need not compare them to humans any more so you should better stop trying to do so and to do as if, just to play act hoping more Elo would sell any better, these days were gone at the latest from the arising of freeware as well as the commercial one as for Elo, that was the rising of Elosion and Celolitis and the end of comparing engines just in this one and only way.

Dr Peter,

i'm not sure i have understood what you meant, so i can say that, giving to an engine an already prepared book is like setting a 100 meters race and starting it from the 30th meter...

If i want to measure the ability of a chess engine i want to see how it treats all kind of positions and all phases of a game: opening, middlegame, endgame.

I cannot make 10.000 games from the start positions because the positions would repeat.

I need to change the start position, but i have to limit more than i can the number of plies that i insert.

Chess960 gives me 960 different start positions and this is the best set i can use.

If i want to have more positions, i just have to select 2 half moves from each of these 960 positions and i would have a lot of sample to test an engine and i think the test would be reliable.

Sorry if i missed something that you meant.

Best Regards

[peter]

If i want to measure the ability of a chess engine i want to see how it treats all kind of positions and all phases of a game: opening, middlegame, endgame.

Then you have to let it play from opening, middlegame and endgame positions.
Otherwise you will always test its abilitiy to play the opening, starting from opening positions, more than the other abilities.
The engine that wins the game already in the opening doesn't have to proof to be as strong in the middlegame and endgame as the one it's playing against.
That's what I meant

[Don]

If i want to measure the ability of a chess engine i want to see how it treats all kind of positions and all phases of a game: opening, middlegame, endgame.

Then you have to let it play from opening, middlegame and endgame positions.
Otherwise you will always test its abilitiy to play the opening, starting from opening positions, more than the other abilities.
The engine that wins the game already in the opening doesn't have to proof to be as strong in the middlegame and endgame as the one it's playing against.
That's what I meant

What we want for Komodo is the ability to play the entire game. No engine can play well that has a serious weakness in any stage but it's clear that some engines play some parts of the games better than others - nothing wrong with that, it is normal. For example we believe that Houdini is particularly strong in the tactical middlegames.

If you are measuring overall strength, you would not start from a book that takes you to the endgame, that is not representative of how a real game is played. A real game is playing from the opening position, right? Last time I checked it was that way. So any opening book for automating testing is a necessary evil designed to provide randomness so that you don't have 50,000 games played the same way.

Since you have no choice but to have a book. at least you can have a book that tries to get out of the way as soon as possible. That is what we have done with Komodo. Of course if you don't care about the opening or want to depend solely on some external book, you would test an engine from random middlegame positions.

There is also the matter of how you define the opening phase too. It's my opinion that even if you are not "technically" out of the opening for all practical purposes many autotest opening books make all the most critical opening decisions for the program and you "essentially" start from the middlegame.

Some people may view the opening as irrelevant since it is possible to obtain comprehensive opening books (which can be considered part of the game playing system) and that is a legitimate point of view - however it's easy to get a program out of the opening with a few non-standard but not ridiculous moves in which case the program will be on it's own and will need to be able to play the opening well.

Is Komodo particularly good compared to other programs (such as Houdini?) Right now it's just a hypothesis - at least for me. I will construct some tests to prove or disprove this. I think Larry believes it is true since he has spent an enormous amount of time getting familiar with Komodo for the book he published and he believes Komodo is the best program of all for opening book preparation.