Perfect X beta3.ctg - little test

[Rubinus]

Little test Perfect X beta3.ctg by Sedat Canbaz:

Test book (and hash) 10m+10s, AMD Sempron 3000+, Fritz 10, 345TB

Rybka 2.3.2a/32bit (128MB hash) - Rybka 2.3.2a/32bit (64MB hash) 41.5:38.5
games:
1-10 R128-R64, RybkaII.ctg-Perfect_X_beta.ctg 4.5:5.5 (0:1=9)
11-20 R128-R64, Perfect_X_beta.ctg-RybkaII.ctg 6.5:3.5 (3:0=7)
21-30 R64-R128, Perfect_X_beta.ctg-RybkaII.ctg 6.0:4.0 (2:0=8)
31-40 R64-R128, RybkaII.ctg-Perfect_X_beta.ctg 5.0:5.0 (0:0=10)
41-50 R64-R128, LoopPH-Book.ctg-Perfect_X_beta.ctg 4.0:6.0 (1:3=6)
51-60 R64-R128, Perfect_X_beta.ctg-LoopPH-Book.ctg 5.5:4.5 (2:1=7)
61-70 R128-R64, Perfect_X_beta.ctg-LoopPH-Book.ctg 5.5:4.5 (1:0=9)
71-80 R128-R64, LoopPH-Book.ctg-Perfect_X_beta.ctg 5.5:4.5 (1:0=9)
=> RybkaII.ctg-Perfect_X_beta.ctg 17.0:23.0 (0:6=34) GRATS!
=> LoopPH-Book.ctg-Perfect_X_beta.ctg 18.5:21.5 (3:6=31) GRATS! (some little repairs after game 60)
http://www.chesspraga.cz/engine-turn/te ... ndbook.rar

90m+30s, Athlon 3200+, 256MB hash, 345TB, Fritz 10 GUI
Fruit 2.3.1 - Shredder 11.1UCI (book by Sedat)
http://www.chesspraga.cz/engine-turn/Fr ... _Sedat.rar
6x ph-gambitbook.bin (cca 4/2007), 6x ph-exoticbook.bin (cca 4/2007), 10x glaurung-ph-book2beeta8.bin (15.8.2007), 8x ph-book.bin (cca 4/2007)
If normally position, new Perfect is very good book.

Spike 1.2 - Shredder 11UCI (+book by Sedat)
http://www.chesspraga.cz/engine-turn/Sp ... _Sedat.rar. But Spike 1.2 very less than Shredder 11.

[Graham Banks]

I tried to split the flaming and personal antagonism from this thread, but my efforts took everything except Pavel's original post!

I apologise to those of you who made some useful remarks amongst the unacceptable stuff.

At present the mods don't have the capacity to restore individual posts.

Regards, Graham (moderation)

[Rubinus]

I tried to split the flaming and personal antagonism from this thread, but my efforts took everything except Pavel's original post!
I apologise to those of you who made some useful remarks amongst the unacceptable stuff.
At present the mods don't have the capacity to restore individual posts.
Regards, Graham (moderation)

Uff, my other posts ...
Best wishes, Graham

[Ovyron]

At present the mods don't have the capacity to restore individual posts.

Isn't it a flaw on the forum's software? Maybe this means that we should get a better one.

[Eelco de Groot]

I tried to split the flaming and personal antagonism from this thread, but my efforts took everything except Pavel's original post!
I apologise to those of you who made some useful remarks amongst the unacceptable stuff.
At present the mods don't have the capacity to restore individual posts.
Regards, Graham (moderation)

Uff, my other posts ...
Best wishes, Graham

Hi Pavel,

Some of these posts are probably still visible if you look with Google but only in their cache. For instance I Googled Pavel H?se Perfect X beta3.ctg. Some of the posts from other authors may still be there also
These are contents of two of your other three posts I could find, but now that they are gone from the forum, I don't know if the Google cache will still keep these much longer:

Post subject: Re: Perfect X beta3.ctg - little test Posted: Tue Dec 04, 2007 1:37 pm

Sedat Canbaz wrote:
Hello Pavel,
Thanks fot the book test...
I checked the games and noticed that there are many unnusual openings !
Regards,
Sedat


Yes, but *.ctg format is troublesome - self-knowledge and impossible pre-set likely variant.
Color for moves -> not for move, but line position!. Set (self-knowledge) value is only half solution.

Example 1:
1.d4 d5 2.e4?!
Best move? -> 2.-de4! But in *.ctg format problem, 2.-e6 and 2.-c6. (games with french and caro-kann more than blackmar-diemer)
Example 2:
1.e4 e5 2.Nf3 Nf6 3.Bc4?!
Best move? -> 3.-Nxe4. *.ctg? -> 3.-Nc6 (more games two knights openings)

For unnusual opening better binary format, or spike format, than *.ctg.
For binary:
compilation after halfmoves level -> number games and results ensure likely variants
For Spike *.scb:
SpikeBookBuilder - manual set.

If use import games, also grandmasters making mistake, specially in gambits -> think blocks
Example:
David - Pinter, France 1993 (source Jansa:"Dynamika strategie zahájení" /Dynamic Strategy Openings)
1.e4 e5 2.Nc3 Nc6 3.f4 ef4 4.Nf3 g5 5.d4 g4 6.Bc4!? (!? -> ) gf3 7.0-0 d5 (??)
ZappaMexico (32bit, Sempron 3000+, cca 1s) 7.-Nxd4! -> If 8.Dxd4, then 8.-Dg5! 9.Rf2, Bc5 )

Post subject: Re: Perfect X beta3.ctg - little test Posted: Mon Dec 03, 2007 11:42 am

For all
Very little test for reputable results!
I find errors, not compare skill (new book by Sedat is very good, but higher or less than other? Min. 1000 games needed, more engines ...). Once I thought that Perfect (and H+S item) not powerful books (compare from Takker, Rybka and RybkaII), but today yes.

Great riot for nothing .

Test book (and hash) 10+10, AMD Sempron 3000+, Fritz 10, 345TB

Rybka 2.3.2a/32bit (128MB hash) - Rybka 2.3.2a/32bit (64MB hash) 35.5:44.5 (11:19=50)
games:
1-20 R128-R64, PH-Gambitbook.ctg(cca 04/2007) - Perfect_X_beta.ctg(v.4) 9.0:11.0 (2:4=14)
21-40 R128-R64, Perfect_X_beta.ctg(v.4) - PH-Gambitbook.ctg(cca 04/2007) 12.5:7.5 (6:1=13)
41-60 R64-R128, Perfect_X_beta.ctg(v.4) - PH-Gambitbook.ctg(cca 04/2007) 14.5:5.5 (11:2=7)
61-80 R64-R128, PH-Gambitbook.ctg(cca 04/2007) - Perfect_X_beta.ctg(v.4) 11.0:9.0 (3:1=16)
=> PH-Gambitbook.ctg(cca 04/2007) - Perfect_X_beta.ctg(v.4) 33.0:47.0 (8:22=50)
(my gambitbook is definitely for try other book, not tournament play - some variants very incorrect)

[Graham Banks]

Hi Pavel,

Some of these posts are probably still visible if you look with Google but only in their cache. For instance I Googled Pavel H?se Perfect X beta3.ctg. Some of the posts from other authors may still be there also
These are contents of two of your other three posts I could find, but now that they are gone from the forum, I don't know if the Google cache will still keep these much longer:

Hi Eelco,

the posts are still in the moderation archive. I'm happy to quote all the relevant posts that weren't part of the flaming if anybody asks.

Regards, Graham.

[Eelco de Groot]

Hi Eelco,

the posts are still in the moderation archive. I'm happy to quote all the relevant posts that weren't part of the flaming if anybody asks.

Regards, Graham.

Hi Graham,

Personally I think that would be preferable to losing the whole thread. It was interesting to read what some of the best bookmakers had to say, and they don't write here a lot. There goes a lot of work in making those opening books, so a bit of rivalry is to be expected, it goes with the territory and I don't think they meant it all bad, at least the earlier parts that I have seen I saw no really insulting stuff? I had not read the posts from the last two days though...

If splitting the thread does not work, -maybe the thread was too long already?, quoting and leaving out the posts that people may have protested against is maybe the next best thing. Or maybe just add a post asking the posters to keep it civil and replace the original thread? You can always lock the thread if it threatens to get out of hand again.. Not so easy to decide I suppose. I don't know really what were these contentious posts... That decision must be up to you Graham!


Regards, Eelco

[Graham Banks]

Posted by Harry Schnapp:

Settings for Rybka.bkt
Extras<Options<Book Options.
Use Opening Book
Book Mode = Tournament.

Requirements:
Games =1
Wins =1
White's Score and Black's Score = default
Move No. = 40
Check mark on: Move Values,Use Statistics and Refutations.
Classic-------------Modern = default
Move Value----------------to + maximum.
Num. Games -------------to + maximum
Score : off-------------------------

These are exactly the settings offered by Vasik Rajlich to download
with Rybka.bkt, the Official Rybka book for Shredder Classic
interface.

Posted by Sedat Canbaz:

Hello Harry,

In my 10 years computerchess life,i have run many tournaments under many different GUIs and i am 100% sure that your book was tested at that (crazy) book option:
http://sedatchess.com/shredder_book.JPG

Let's make it more clear,my request to Pavel in my book testing was not to show that your books are weak,just i wanted to be tested by a neutrlal tester,becouse there were some suspisous about the performance of my beta book on Rybka forum:
http://rybkaforum.net/cgi-bin/rybkaforu ... t#pid30894

And Pavel's test for me its very important,in this way i can see my book weakness and then to update before final release !

There is no doubt that your books are one of the srongest ones for Rybka playing style,thats why i said to Pavel to choose as a oponent your book !

Just i dont uderstand you,why dont wish to test against your books ?!
Its not a oficial tournament,just a litle book test,for example with a such book testing we can see our book move weaknesses and update in the next releases !

Only i dont agree with that text of you:
A prominent computerchess man has written here :" book tournaments
are brainless olympiad". I frame less hard: "random circus".
so thats means all users on playchess server,which play mostly Rybka versions between each other with different books are the games brainless ???


Dont worry,if you check carefully the results there is no big diferenses in elo between top 3 books:

Code: Select all

Rank Name                     Elo    +    - games score oppo. draws

   1 Rybka Perfect X b2.ctg  3062   19   19   720   57%  3020   67%
   2 Rybka RybkaII.ctg       3057   12   12  2060   60%  2998   59%
   3 Rybka Rybka.bkt         3056   24   24   450   56%  3018   61%
   4 Rybka Perfect X b1.ctg  3049   14   14  1280   55%  3018   65%
   5 Rybka Sheebar.ctg       3043   17   17   900   53%  3026   63%
   6 Rybka 2750test1.ctg     3043   18   18   780   51%  3037   66%
   7 Rybka TourBookII.ctg    3036   32   32   250   54%  3010   59%
   8 Rybka HS-for-Rybka.ctg  3033   15   15  1180   52%  3024   66%
   9 Rybka Xmas2640-12.ctg   3033   26   26   370   51%  3025   68%
  10 Rybka Hiarcs10.ctg      3027   32   32   230   52%  3016   71%
  11 Zappa Perfect X b2.ctg  3017   23   23   470   45%  3046   59%
  12 Rybka Paradigm.ctg      3015   35   35   210   55%  2982   55%
  13 Rybka Remis.ctg         3010   32   32   240   50%  3012   65%
  14 Rybka DFritz10.ctg      3008   32   32   230   50%  3007   68%
  15 Rybka Junior10.ctg      3002   32   32   230   49%  3007   68%
  16 Rybka TheTurkII.ctg     2994   32   32   230   45%  3021   70%
  17 Zappa Perfect X b1.ctg  2991   26   26   400   41%  3042   55%
  18 Rybka General.ctg       2987   33   33   230   44%  3022   61%
  19 Zappa Sheerbar.ctg      2983   23   23   500   39%  3049   51%
  20 Zappa RybkaII.ctg       2981   35   35   200   46%  3005   62%
  21 Zappa DFritz10.ctg      2962   35   36   200   41%  3014   60%
  22 Zappa Junior10.ctg      2959   36   36   200   41%  3014   56%
  23 Zappa Zap!Chess.ctg     2951   36   37   200   35%  3036   57%
  24 Zappa TourBookII.ctg    2939   37   38   200   33%  3044   48%
  25 Zappa HS-Book.ctg       2931   27   28   400   33%  3041   40%
  26 Zappa Paradigm.ctg      2920   38   39   200   30%  3045   42%
  27 Zappa Shredder9.bkt     2909   32   33   300   28%  3059   35%

Best,
Sedat

Posted by Harry Schapp:

Hello Sedat,
I have 28 years experience in computerchess.
In my bad english I will describe some points why "book tournaments"
are worthless. If you have 10 good books and repeat 5 times the same tournament,there will always give huge shifts.The number 1 will be number 7,the number 10 will be number 3,etc..etc..
Another important example: Book A comes later out of book,wins also time and has also a better position which is kept for long time.And in the middlegame or endgame(move 28 or 32 or 54 etc)the engine with the book A makes error and loses.What does that have to do with the book?
Then is the book "B" better than book "A" ??. I have made for years hundrets such comparisons and noted that such cases often appear.
Only when a book is SIGNIFICANTLY better than another,can be precisely specified.On this subject prior years,I have reported with a lot examples in the best German chess magazine.
I have else some arguments, but I think that's enough.
Best regards to all young chess friends from the veteran (73)
Harry

Posted by Sedat Canbaz:

Hello Harry,

Thanks for that useful information...

Just i'd like to add:
-for creating a very strong opening book,it needs a lot of testing,thousand of games with different time controls against the strongest available books,
plus all book move lines must be handly optimized to find the strogest move,on loosing disadvantage human openings must be anylized by the strongest engines and then to fix a different stronger book move !

In other words,before to release X book,it needs a lot of human efforts !

And after all,with a big possibility it will be ranked at the top 3 strongest books !!!


Best Regrds,
Sedat Canbaz

Posted by Albert Silver:

The issue of books is extremely complicated, and truly you can't possibly compare one book over another in terms of quality without a huge sample number.

To begin with, a book's results will vary simply according to the engine's style. In fact, if you test the exact same position from a book, such as the opening suites (Nunn, Silver, Noomen) much less the entire book, and make a single small change in an engine's parameters, you will get a different game and result, against a same opponent. So imagine an entire book.

I think that a well-tuned book is as much an exercise in removing bad lines, as it is in finding ones that a particular engine handles best, but as to proving which is better... forget it.

Enjoy the books, and the games, but don't waste time trying to prove which is best, unless you are prepared to run at least 20,000 games (probably more) to get any kind of statistically relevant information.

Just my 2 cents.

Albert

Posted by Nelson Hernandez:

A puzzling discussion. It seems pretty obvious to me that measuring book performance based on game outcomes is nearly useless. The book's primary job is to get you to an advantageous position from which the engine takes over. It's secondary job is to give you an advantage on the clock, which is effectively the same as clocking down your opponent's machine.

So my solution from the very start has been to measure performance based on objective criteria that only measure those two factors. It's very tedious work doing it by hand, one game at a time, plugging numbers to a spreadsheet.

Based on my own private analysis Harry's books are almost unparalleled in their ability to arrive at decent positions with remarkable consistency. What makes his books even more remarkable is that they are so relatively small. Packing so much muscle into a small book is a feat of legendary German craftsmanship.

My conclusion is that Harry is wrong--book tests are VERY useful and CAN measure relative book quality using imperfect but consistent criteria. Nothing like 20,000 games are needed to arrive at a good conclusion (I prefer 500 games to feel confident in the most important measures; 200 will do in a pinch). But Harry is right that Sedat's tournaments aren't the way to get at the truth. They're fun and I like reading about them, but they mean practically nothing.

Posted by Ulysses P:

Yes, but it must be measured on a games' outcome basis. It doesn't matter if the engine leaves book with a +0.90 score if it doesn't have any idea of what is it doing and loses.

I'm of the opinion that Zappa beat Rybka due to this (Not that Rybka was bad out of book, but that she was obviously playing positions that she didn't understood and went to lose, it doesn't matter how good is a position if the engine doesn't get to play it well).

A book maker should make sure to tweak a book for the engine's needs, that's why I don't trust generic books.

Posted by Nelson Hernandez:

What you are describing is a statistical outlier. For the purposes of my analysis, which as I said uses imperfect measures on a consistent basis, the fundamental truth is that there is a direct and fairly high positive correlation between book-exit evaluation and game outcome. Obviously this was one of the first things I measured and considered.

I don't ignore game outcomes. I am only saying that they are a far less reliable and descriptive measure of what the *book* achieved.

Posted by Harry Schapp:

Hello Nelson,
Many thanks. I read and appreciate your very reasoned postings in
the CCC and Rybka forum.

Best regards,
Harry

Posted by Nelson Hernandez:

First of all, I collect ALL the games you post, Sedat. You might say I am a long-time, loyal customer. ALL of your games are worth collecting and I am a regular visitor to your 'chok iyi' website. If you are hiding any more games on your hard drives, I want them!

Second, my remarks referred only to your book tournaments. As for-fun-only tournaments they are fine and I don't see any reason not to run them. Harry is only upset because they are not sufficiently scientific and thus the conclusions that some might infer from the tournament results might be wildly at variance with the truth. This is not because you are malicious or mendacious. It is because the measurements being taken are not really the most relevant ones. Unfortunately, the interesting measurements require a lot of dull, repetitive, painfully labor-intensive work which I wish some hero would come along and automate.

Posted by Sedat Canbaz:

Nelson,

Chok Teshekurler for your kind words...

Best,
Sedat

Posted by Pavel Haase:

For all
Very little test for reputable results!
I find errors, not compare skill (new book by Sedat is very good, but higher or less than other? Min. 1000 games needed, more engines ...). Once I thought that Perfect (and H+S item) not powerful books (compare from Takker, Rybka and RybkaII), but today yes.

Great riot for nothing Shocked Very Happy .

Test book (and hash) 10+10, AMD Sempron 3000+, Fritz 10, 345TB

Rybka 2.3.2a/32bit (128MB hash) - Rybka 2.3.2a/32bit (64MB hash) 35.5:44.5 (11:19=50)
games:
1-20 R128-R64, PH-Gambitbook.ctg(cca 04/2007) - Perfect_X_beta.ctg(v.4) 9.0:11.0 (2:4=14)
21-40 R128-R64, Perfect_X_beta.ctg(v.4) - PH-Gambitbook.ctg(cca 04/2007) 12.5:7.5 (6:1=13)
41-60 R64-R128, Perfect_X_beta.ctg(v.4) - PH-Gambitbook.ctg(cca 04/2007) 14.5:5.5 (11:2=7)
61-80 R64-R128, PH-Gambitbook.ctg(cca 04/2007) - Perfect_X_beta.ctg(v.4) 11.0:9.0 (3:1=16)
=> PH-Gambitbook.ctg(cca 04/2007) - Perfect_X_beta.ctg(v.4) 33.0:47.0 (8:22=50)
(my gambitbook is definitely for try other book, not tournament play - some variants very incorrect)

Posted by Sedat Canbaz:

Hello Pavel,

Thanks fot the book test...

I checked the games and noticed that there are many unnusual openings !

Regards,
Sedat

Posted by Pavel Haase:

Yes, but *.ctg format is troublesome - self-knowledge and impossible pre-set likely variant.
Color for moves -> not for move, but line position!. Set (self-knowledge) value is only half solution.

Example 1:
1.d4 d5 2.e4?!
Best move? -> 2.-de4! But in *.ctg format problem, 2.-e6 and 2.-c6. (games with french and caro-kann more than blackmar-diemer)
Example 2:
1.e4 e5 2.Nf3 Nf6 3.Bc4?!
Best move? -> 3.-Nxe4. *.ctg? -> 3.-Nc6 (more games two knights openings)

For unnusual opening better binary format, or spike format, than *.ctg.
For binary:
compilation after halfmoves level -> number games and results ensure likely variants
For Spike *.scb:
SpikeBookBuilder - manual set.

If use import games, also grandmasters making mistake, specially in gambits -> think blocks
Example:
David - Pinter, France 1993 (source Jansa:"Dynamika strategie zahájení" /Dynamic Strategy Openings)
1.e4 e5 2.Nc3 Nc6 3.f4 ef4 4.Nf3 g5 5.d4 g4 6.Bc4!? (!? -> ) gf3 7.0-0 d5 (??)
ZappaMexico (32bit, Sempron 3000+, cca 1s) 7.-Nxd4! -> If 8.Dxd4, then 8.-Dg5! 9.Rf2, Bc5 )